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EXECUTIVE  SUMMARY 


The  Separation  and  Control  Hiring  Assessment  (SACHA)  program  has  the  goal  of  developing  a 
valid  selection  process  (i.e.,  test  battery)  for  selecting  air  traffic  controllers.  Component  tasks  of 
this  program  include  job  analysis,  predictor  development,  criteria  development,  and  validation  of 
these  predictors  and  criteria.  Ideally,  these  performance  criteria  would  be  based  on  measures 
taken  from  the  controller's  own  sector.  However,  a  controller's  performance  may  vary  depending 
on  the  amount  of  time  he  or  she  has  spent  working  the  sector.  In  addition,  sectors  vary  in 
complexity,  and  therefore,  in  difficulty  for  the  controller.  A  standard  generic  sector  could  be  a 
solution  in  that  the  conditions  under  which  performance  is  measured  would  be  the  same  for  all 
participants.  This  would  be  a  significant  advantage  over  using  performance  measured  on  each 
controller's  home  sector  where  many  factors  such  as  familiarity  and  sector  complexity  vary 
unsystematically.  The  question  remains  concerning  the  validity  of  measurement  based  on  generic 
airspace.  Specifically,  will  a  controller's  performance  in  a  generic  airspace  be  representative  of, 
and  related  to,  performance  achieved  at  his  or  her  home  sector. 

The  generic  sector  evaluated  in  this  study  was  based  on  a  four-comer  post  operation  typically 
used  in  many  terminal  areas  in  the  United  States.  Arrival  aircraft  originated  from  one  of  four 
arrival  fixes  just  outside  the  sector  boundaries.  These  arrival  routes  can  be  thought  of  as  spokes 
of  a  wheel  with  the  main  airport  site  as  the  hub.  In  addition  to  the  main  airport,  there  were  three 
satellite  airports  that  were  under  radar  control.  Departure  aircraft  from  the  main  and  satellite 
airports  were  sent  directly  to  one  of  four  departure  fixes  located  outside  the  sector  boundaries. 

Eleven  air  traffic  controllers  from  the  Atlantic  City  International  Airport  (ACY)  Terminal  Radar 
Approach  Control  (TRACON)  participated  in  the  study.  The  experiment  was  conducted  at  the 
Federal  Aviation  Administration  Technical  Center's  Human  Factors  Laboratory  at  Atlantic  City 
International  Airport,  New  Jersey. .  The  experimental  apparatus  consisted  of  a  high  fidelity  air 
traffic  control  (ATC)  simulator  with  voice  communication  equipment  to  allow  controllers  to  issue 
commands  to  remote  simulation  pilots.  Each  controller  performed  nine  different  scenarios  over  2 
days  of  testing.  The  first  day  of  testing  was  considered  a  training  day  where  controllers 
performed  one  low  traffic  volume  mn  on  the  ACY  sector  and  then  four  mns  on  the  generic 
sector.  These  generic  mns  were  of  moderate  traffic  volume.  The  second  day  was  considered  a 
test  day  where  controllers  performed  four,  1-hour  mns.  Two  (one  low  volume,  one  high  volume) 
of  these  were  on  the  home  sector  and  two  (one  low  volume,  one  high  volume)  were  on  the 
generic  sector.  Low  volume  mns  consisted  of  7  aircraft  appearing  every  15  minutes,  moderate 
traffic  mns  used  for  training  consisted  of  10  aircraft  appearing  every  15  minutes,  and  high  traffic 
mns  consisted  of  1 1  aircraft  appearing  every  15  minutes. 

Data  reflecting  ATC  performance,  workload,  system  effectiveness,  and  self-assessment  of 
performance  were  collected  during  the  simulation.  Some  additional  controller  performance 
measures  were  collected  using  a  new  over-the-shoulder  rating  form  in  development  for  the 
SACHA  program.  Dimensions  on  this  form  included  communication  and  informing,  managing 
multiple  tasks,  maintaining  attention  and  vigilance,  and  maintaining  a  safe  and  effective  traffic 
flow.  System  effectiveness  measures  included  number  of  controller  transmissions,  aircraft  density, 
and  number  of  clearances  issued.  Controller  workload  was  assessed  using  the  Air  Traffic 
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Workload  Input  Technique  (ATWIT).  The  ATWIT  consisted  of  collecting  participants'  ratings  of 
workload  as  they  controlled  traffic. 

In  addition  to  the  previously  described  measures,  several  questionnaires  were  used  to  collect 
subjective  ratings  from  participants.  First,  a  demographic  questionnaire  was  completed  that 
requested  background  information  from  each  participant.  In  addition,  after  each  scenario  was 
finished,  controllers  made  self  assessment  ratings  of  their  own  performance  in  a  post-scenario 
questionnaire.  A  final  questionnaire  was  administered  at  the  end  of  the  simulation  to  obtain 
subjective  impressions  of  the  realism  of  the  simulation  and  the  representativeness  of  the  generic 
sector. 

The  results  showed  significantly  lower  ATWIT  ratings  by  the  last  generic  run  compared  to  the 
first  generic  run  on  the  first-day  training  runs.  Time  under  control  and  the  distance  flown  by  the 
aircraft  significantly  decreased  by  the  last  training  run.  Controller  ratings  of  workload  and  stress 
were  also  significantly  lower  by  the  last  generic  run.  In  addition,  post-scenario  questionnaire 
ratings  for  ability  to  plan,  exchange  information,  and  prioritize  were  significantly  higher  by  the  last 
run. 

Correlation  co-efficients  between  scores  on  the  generic  sector  and  scores  on  the  ACY  sector  were 
significant  for  the  over-the-shoulder  ratings,  ATWIT  ratings,  and  post-scenario  questionnaire 
ratings.  The  correlation  co-efficients  were  higher  and  most  consistent  for  the  high-volume  traffic 
runs.  The  correlation  co-efficients  for  the  system  effectiveness  variables  were  moderate  and  not 
consistent  across  variables.  However,  when  scores  from  the  high  and  low  traffic  runs  were 
averaged  and  then  correlated  between  sectors,  most  of  the  correlation  co-efficients  increased. 
These  runs  were  averaged  because  the  combination  of  runs  provided  a  larger  sample  of  the 
controller's  performance  than  either  run  separately.  In  addition,  the  combination  of  high  and  low 
runs  provided  a  data  point  which  reflected  performance  over  a  range  of  traffic  volume.  The 
results  suggest  that  more  runs  are  needed  to  obtain  significant  correlation  co-efficients  for  the 
system  effectiveness  variables. 

Final  questionnaire  comments  indicated  that  the  participants  thought  the  simulation  was  very 
realistic.  In  addition,  all  participants  thought  the  hands-on  training  for  the  generic  sector  was  very 
adequate.  The  majority  of  the  controllers  thought  the  generic  sector  was  representative  of  a 
typical  terminal  environment.  Most  controllers  expressed  that  they  had  a  positive  experience 
working  on  the  project. 

The  fact  that  performance  indicators  did  not  change  appreciably  over  the  four  training  runs  is 
considered  a  positive  finding.  Genera  was  designed  for  ease  of  implementation.  Controllers  were 
able  to  learn  it  very  quickly  and  work  traffic  with  no  major  complications.  Ideally,  a  generic  sector 
should  not  pose  hurdles,  but  rather  facilitate  performance,  as  was  accomplished  by  this  sector. 
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1  INTRODUCTION. 


1.1  PROBLEM  STATEMENT. 

Since  the  beginning  of  human  kind,  people  have  evaluated  each  other's  performance.  Much  of  this 
evaluation  was  based  on  individual  standards  of  which  the  evaluator  may  not  have  even  been 
aware.  If  one  asks  an  expert  in  any  field  what  constitutes  "good"  performance,  he  or  she  may 
provide  an  answer  that  has  meaning  for  them  alone,  and  the  response  may  or  may  not  translate  to 
the  expectations  of  another  expert. 

A  meeting  was  convened  in  June  1987  by  the  Federal  Aviation  Administration's  (FAA’s)  Air 
Traffic  Requirements  (ATR)  organization  to  discuss  the  nature  of  air  traffic  controller  errors,  and 
their  impact  on  operations  and  training.  It  was  noted  that  while  automation  had  increased,  the 
number  of  aircraft  that  a  single  controller  could  work  had  not  increased  appreciably,  and 
controllers  continued  to  make  the  same  sort  of  mistakes.  These  were  often  attributed  to  a  failure 
to  perceive  critical  information. 

Lauber  (1993,  p.  23)  expressed  his  concern  about  human  performance  issues  in  Air  Traffic 
Control  (ATC).  He  stated  that  "human  performance  issues  clearly  present  the  major  challenge  to 
all  of  us.  If  we  are  to  make  the  system  significantly  safer  we  must  find  ways  of  minimizing  or 
eliminating  all  together,  human  error  induced  accidents.” 

In  1993,  there  were  764  controller  operational  errors  in  the  United  States  (FAA,  1994),  a  slight 
increase  from  738  errors  the  previous  year.  The  FAA  is  constantly  trying  to  reduce  the 
probability  of  these  errors.  By  developing  more  effective  measurement  tools,  it  may  be  possible 
to  better  understand  the  true  range  of  acceptable  performance. 

1.2  ASSUMPTIONS  AND  GOALS. 

When  novices  are  taught  a  skill,  they  are  trained  in  one  of  two  ways.  One  is  based  on  an  absolute 
standard  of  performance  that  is  clearly  defined  in  advance  and  easily  recognized  by  anyone  in  the 
trade  or  occupation.  The  other  method  is  to  use  a  relative  standard  that  is  based  on  how  everyone 
else  does  or  on  the  trainers'  understanding  of  what  it  takes  to  perform  the  task  set  (Berlinger, 
Angell,  &  Shearer  1964).  In  this  second  situation,  the  training  system  is  very  much  dependent  on 
the  trainer  and/or  on  how  all  the  other  trainees  are  doing.  These  relative  standards  make 
performance  measurement  very  complicated.  There  has  also  been  a  lack  of  integration  between 
training  theory  and  evaluation  models  for  complex  performance  (Cannon-Bowers,  Tannenbaum, 
Salas,  &  Converse,  1991). 

In  ATC,  there  are  some  absolute  or  minimum  standards  against  which  everyone  is  judged.  One  of 
the  most  fundamental  standards  is  based  on  the  minimum  separation  allowed  between  aircraft 
under  positive  radar  control.  Every  controller  must  achieve  this  if  he  or  she  is  to  stay  in  the 
system.  Since  this  is  an  absolute  standard,  everyone  who  lasts  in  ATC  meets  it  or  risks  being 
removed.  This  means  that  this  standard  is  not  very  useful  for  looking  at  the  range  of  performance 
that  controllers,  as  all  human  operators,  produce.  The  system  has  evolved  into  the  use  of  relative 
standards  employing,  as  the  basic  metric,  an  over-the-shoulder  rating  scale  that  is  open  to 
considerable  latitude  in  interpretation  (FAA,  1990).  Each  evaluator  must  introduce  his  or  her 
experience  and  biases  when  doing  a  controller  check  ride  or  evaluation. 
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Performance  is  a  complex  construct  that  has  seen  considerable  research  over  the  years.  While 
there  are  many  definitions  of  performance,  the  following  is  an  operational  definition  that  is 
currently  being  used  in  the  research  to  be  discussed  in  this  report: 

Performance  is  the  accomplishment  of  a  task  or  interrelated  set  of  tasks  in 
relation  to  a  defined  and  specified  standard  while  operating  within 
constraints  of  space,  time,  and  resources. 

This  definition  means  that  a  human  operator  is  involved.  The  operator  must  accomplish 
something  in  relation  to  a  specified  standard.  If  the  behavior  exceeds  the  standard,  it  is  evaluated 
as  successfiil,  and  if  it  fails  to  meet  the  standard,  it  is  not  successful.  The  distance  above  or  below 
the  standard  determines  different  levels  of  accomplishment  within  the  unsuccessful  and  successful 
categories  respectively. 

This  current  experiment  is  based  on  the  assumption  that  the  performance  of  air  traffic  controllers 
can  be  measured  in  a  number  of  ways.  It  is  also  based  on  the  belief  that  the  quality  of  this 
measurement  can  continue  to  be  improved,  and  that  this  improvement  is  a  worthwhile  endeavor. 

This  research  in  controller  performance  is  being  done  for  a  number  of  reasons.  First,  it  is  a  step  to 
improve  performance  and  reduce  the  possibility  for  error.  Second,  it  evaluates  the  feasibility  of 
using  airspace  models  for  testing  and  training  that  the  participating  controllers  have  not  seen 
before  and  have  not  over-learned  with  practice.  The  use  of  generic  airspace  can  simplify  and 
reduce  the  cost  of  training  and  selection  if  personnel  are  able  to  perform  as  well  with  it  as  they  can 
with  an  over-learned  environment. 

This  experiment  is  also  a  stepping  stone  for  a  follow-on  effort  that  will  use  video  tapes  collected 
during  the  experiment.  The  follow-on  project  will  cross  validate  measures  collected  in  the 
simulation,  to  be  described  in  a  latter  section  of  the  report.  The  cross  validation  shall  examine  the 
relationship  of  the  simulation  measures  to  those  collected  from  supervisory  or  training  controllers, 
who  will  evaluate  the  performances  that  they  see  on  the  video  tape. 

1 .3  REVIEW  OF  THE  RELATED  LITERATURE. 

Even  when  performance  standards  are  absolute,  clear  definitions  have  to  be  agreed  upon 
concerning  the  desirable  behavior.  When  the  standards  are  relative,  they  depend  very  heavily  on 
the  trainer's  internalized  model  of  what  good  performance  is  all  about. 

Rault  (1979)  pointed  out  that  flight  crew  personnel,  for  example,  often  establish  an  operational 
standard  against  which  they  compare  their  own  performance.  They  tend  to  judge  themselves  in 
relationship  to  how  closely  their  performance  resembles  this  standard.  It  is  likely  that  most 
professionals  operating  in  high  reliability  organizations  have  both  minimum  external  standards  and 
internalized  personal  standards.  Warm  and  Dember  (1986)  expressed  concern  about  the  level  of 
alertness  of  personnel  who  operate  complex  systems  and  spend  much  of  their  time  monitoring 
data  flows.  Even  with  a  great  deal  of  motivation  and  a  fairly  high  internal  standard,  human 
operators  can  lose  their  focus.  In  aviation,  it  does  not  take  much,  in  terms  of  a  loss  of  situation 
awareness  (SA),  to  create  problems. 
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While  human  beings  add  flexibility  and  adaptability  to  the  system,  they  also  add  the  potential  for 
error.  Senders  and  Moray  (1991,  p.  1)  describe,  "All  of  us  have  experienced  human  error.  When 
we  interact  with  machines  or  complex  systems,  we  frequently  do  things  that  are  contrary  to  our 
intentions.  Depending  on  the  complexity  of  the  system  and  the  intentions  of  the  people 
interacting  with  it,  this  can  be  anything  from  inconvenience  (often  it  is  not  even  noticed)  to  a 
genuine  catastrophe." 

In  an  early  comprehensive  study  of  controller  errors,  Kinney,  Spahn,  and  Amato  (1977)  analyzed 
FAA  reports  and  developed  eight  categories  of  errors.  These  included:  controlling  in  another's 
airspace,  timing  and  completeness  of  flight  data  handling,  inter-positional  coordination  of  data, 
use  of  altitude  on  the  display,  procedures  for  scanning  and  observing  flight  data,  phraseology  and 
use  of  voice  communications,  use  of  human  memory  to  include  relying  on  recall  in  a  noisy 
environment,  and  dependence  on  automatic  capabilities. 

Today,  the  FAA  uses  a  different  set  of  categories  to  classify  operational  errors.  In  the  FAA 
(1988),  the  following  categories  were  employed;  radar  display,  communication,  coordination, 
aircraft  observation,  data  posting,  and  position  relief.  By  far,  the  most  frequent  source  of  errors 
identified  was  in  a  subclass  of  "radar  display:  the  misuse  of  data.”  This  category  implies  that 
information  was  available  and  was  either  misinterpreted  or  inaccurately  stored  in  working 
memory. 

Controller  performance  issues  are  not  limited  to  only  a  litany  of  errors  made  in  an  operational 
setting.  Research  has  been  conducted  for  over  25  years  on  various  ways  of  trying  to  quantify 
performance.  McKenzie,  Buckley,  and  Sarlanis  (1979)  conducted  a  study  of  the  potential 
usefulness  of  physiological  indicators  to  evaluate  controller  workload.  In  this  study,  10 
controllers  watched  films  of  a  simulated  radar  displa  and  were  asked  to  identify  potential  aircraft 
conflicts.  These  conflicts  occurred  in  two  counterbalanced  conditions  where  the  aircraft  volume 
differed  considerably.  The  goal  was  to  determine  if  controllers  would  respond  physiologically  to 
the  differing  demands  of  the  two  conditions.  Both  heart  rate  and  galvanic  skin  resistance  (GSR) 
were  measured.  The  results  indicated  that  heart  rate  did  not  discriminate  across  the  two 
conditions,  but  GSR  frequency  changes,  and  the  area  under  the  GSR  plot,  were  significantly 
different  between  high  and  low  system  demands. 

McKenzie  et  al.  (1979)  noted  that  the  scenarios  they  created  were  extreme,  and  physiological 
measures  may  or  may  not  be  sensitive  under  conditions  where  the  differences  in  task  demands  are 
not  so  diverse.  Further,  this  exploratory  study  did  not  involve  the  requirements  that  controllers 
actually  separate  traffic.  They  did  not  have  the  stresses  of  responsibility  that  are  characteristic  of 
the  control  task.  So,  any  conclusions  may  not  necessarily  generalize  to  either  an  operational  or  an 
ATC  simulation  environment. 

Buckley,  O'Connor,  Beebe,  Adams,  and  MacDonald  (1969)  conducted  what  may  have  been  the 
first  simulation  study  of  air  traffic  controllers  that  included  physiological  measurement.  Their 
primary  focus  was  on  the  assessment  of  controller  performance  and  its  relationship  to 
chronological  age.  However,  they  also  collected  two  physiological  indicators:  heart  rate  and 
GSR.  They  found  a  relationship  between  heart  rate  and  heart  rate  variability  and  objective 
measures  of  task  load,  the  average  density  of  aircraft  under  control.  The  correlation  co-efficients 
were  small,  usually  less  than  r=.38,  but  significant.  The  authors  concluded,  "These  results  confirm 


3 


the  hypothesis  that  physiological  functions  may  be  sensitive  indicators  of  workload"  (pp  2-7). 

This  was  one  of  the  studies  that  has  been  conducted  to  examine  controller  performance  and 
workload  issues. 

Systems  designers  are  most  concerned  with  measures  of  primary  task  performance.  This  can  be 
complicated  by  workload  and  other  factors  such  as  task  load,  which  is  the  demand  placed  on  the 
operator  by  the  environment.  When  workload  and  performance  are  measured  separately  across  a 
wide  range  of  task  load,  they  can  be  inversely  related  to  each  other  as  task  load  and  workload 
increase  (Stein,  1985).  However,  when  examining  operational  errors  as  a  performance  indicator, 
there  is  a  common  finding  that  errors  occur  more  frequently  at  lower  to  moderate  levels  of  task 
load  (Rodgers,  1993;  Kinney  et  al.,  1977;  FAA,  1988).  This  finding  has  been  demonstrated  both 
in  the  United  States  and  by  Transport  Canada  (Stager  &  Hameluck,  1990).  Rodgers  and  Duke 
(1993)  suggest  that  previous  taxonomies  of  errors  have  been  incomplete  and  may  have  missed 
information  processing  failures  that  subsequently  led  to  inappropriate  actions.  This  occurs  when 
task  load  is  defined  as  number  of  aircraft  or  in  terms  of  complexity  that  is  assessed  using  a  rating 
scale  1  (low)  to  5  (high)  complexity  (standard  FAA  form  for  operational  error  investigations). 

Seven  (1989)  stated  that  "it  is  in  the  real  world  that  workload  problems  contribute  to  accidents 
and  system  inefficiencies,  and  result  in  over-manning  or  under-manning  on  critical  tasks."  She 
suggests  that  we  develop  unobtrusive  measures  specific  to  the  operational  systems,  and  then 
generate  realistic  data  based  on  real-time  measurement.  In  essence,  she  proposed  the  use  of 
noncritical  tasks  as  indicators  of  workload,  assuming  that  performance  would  decline  as  the  load 
from  primary  tasks  increased.  This  is  basically  similar  to  the  theory  behind  secondary  task 
techniques  without  the  addition  of  artificial  secondary  tasks. 

There  has  also  been  considerable  effort  expended  in  a  search  for  task  or  environmental  models  of 
workload  in  ATC.  While  it  is  generally  believed  that  workload  and  performance  are  related,  the 
nature  of  this  relationship  continues  to  be  disputed.  One  of  the  oldest  environmentally-oriented 
models  of  system  task  load,  which  is  related  to  workload  and,  therefore,  to  performance,  was 
developed  by  Arad  (1964).  He  identified  three  basic  components  of  load;  background,  routine, 
and  airspace.  Arad  created  a  mathematical  model  for  computing  overall  load  based  on  such 
variables  as  aircraft  types,  and  what  they  were  doing  in  the  airspace.  This  modeling  activity  was 
designed  for  use  in  answering  staffing  questions,  rather  than  evaluating  operator  real  time 
activities. 

Jolitz  (1965)  decided  to  conduct  a  comprehensive  test  of  the  Arad  model  using  simulation  of  16 
ATC  sectors  and  testing  the  degree  to  which  the  model  would  predict  mean  subjective  ratings  of 
load  by  controllers.  Jolitz  concluded  that  there  was  no  relationship,  and  that  a  better  predictor  of 
controller's  concept  of  load  was  simply  the  number  of  aircraft  handled  per  hour. 

Robertson,  Grossberg,  and  Richards  (1979)  developed  and  evaluated  another  computer  model  of 
controller  activity  which  had  both  workload  and  performance  implications.  They  referred  to  this 
model  as  the  relative  capacity  estimating  process  (RECEP).  It  places  a  heavy  emphasis  on  system 
events  and  functions  in  an  off-line  data  processor  capable  of  analyzing  these  events  after  they  have 
Occurred.  While  the  primary  purpose  of  the  model  was  to  estimate  workload,  it  did,  by  necessity, 
examine  controller  activities.  It  divided  these  activities  into  three  general  categories;  routine, 
surveillance,  and  conflict  prevention.  These  were  similar  to  those  proposed  by  Arad  (1964).  By 
computing  and  summing  all  the  sub-task  performance  times,  Robertson  et  al.  (1979)  proposed 
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maximum  limits  in  terms  of  man-  minutes-per-hour  of  operational  time.  RECEP  measures 
correlated  favorably  with  subject  matter  expert’s  (SME's)  ratings  of  work  pace.  There  v/as 
considerable  variability  across  different  airspace  sectors. 

One  of  the  strengths  of  the  work  of  Robertson  et  al.  (1979)  was  its  use  of  data  from  operational 
facilities.  There  is  a  more  recent  program  that  makes  even  more  comprehensive  use  of 
operational  data  and  has  a  strong  performance  orientation.  This  program  is  called  the  Situation 
Assessment  though  Recreation  of  Incidents  (SATORI)  and  is  being  developed  by  personnel  at  the 
Civil  Aeromedical  Institute  in  Oklahoma  City,  Oklahoma  (Rodgers  and  Duke,  1993).  SATORI 
analyzes  system  analysis  report  (SAR)  tapes  that  contain  all  the  operation  events  for  one  radar 
position  over  a  given  time  period.  These  tapes  are  routinely  recorded  in  ATC  centers.  The 
original  purpose  of  SATORI  was  to  evaluate  the  factors  that  led  up  to  an  airspace  incident  or 
controller  operational  error.  Rodgers,  Manning,  and  Kerr  (1994)  have  taken  this  project  one  step 
further.  They  have  developed  the  performance  and  objective  workload  research  program 
(POWER).  This  software  package  will  allow  for  the  output  of  many  performance  measures 
described  by  Stein  (1992). 

Controller  performance  measurements  have  consistently  involved  tasks  and  variables  derived  from 
ATC  and  produced  findings  expressed  in  ATC  terms  (Hopkin,  1980).  Another,  possibly  more 
beneficial,  approach  would  be  to  trace  the  origins  of  the  practical  difficulties  (e.g.,  memory  lapses) 
that  the  controller  encounters  to  limitations  in  human  cognitive  capabilities,  and  to  use  basic 
psychological  knowledge  to  explain,  measure,  and  resolve  them.  It  is  fundamental  to  consider  the 
controller's  task  in  human  terms  in  order  to  provide  perspectives,  explanations,  and  insights  into 
the  cognitive  processes  that  support  ATC.  While  the  use  of  new  technologies  may  be  essential  in 
order  to  deal  with  the  ever-increasing,  information-processing  demands  of  the  ATC  system,  the 
long-term  performance  implications  of  extended  use  of  the  new  technologies  on  human 
performance  are  largely  unknown  (Endsley,  1988;  Harwood,  Barnett,  &  Wickens,  1988). 

As  ATC  automation  increases,  more  attention  to  the  fundamental  cognitive  aspects  of  the 
controller's  job  is  necessary.  It  is  necessary  to  develop  cognitive  performance  measures  so  that 
the  consequences  of  automation  on  controller  performance  can  be  effectively  evaluated  (Hopkin, 
1991).  This  implies  that,  in  the  long  run,  we  may  have  .to  expand  the  more  traditional  views  of 
what  performance  is  to  encompass  concepts  that  we  have  viewed  in  the  past  as  unrelated  or 
inconsequential. 

Practice  is  the  single  most  powerful  factor  for  improving  the  controller's  ability  to  perform  ATC 
tasks.  Nothing  is  likely  to  offset  the  frailties  of  working  memory  as  will  practice.  The  influence 
of  practice  on  the  attentional  demands  of  working  memory  has  received  considerable  attention 
within  the  framework  of  automatic  and  controlled  processing  (Schneider  &  Shifftin,  1977). 
Controller  error  rates  have  been  associated  with  the  proportion  of  full  performance  level  (FPL) 
controllers  (those  most  practiced  and  proficient).  Rodgers  (1993)  accomplished  an  analysis  of  the 
FAA's  operational  error  database.  He  found  that  facility  error  rates  were  inversely  proportional  to 
the  percentage  of  the  work  force  that  had  achieved  FPL  status. 

Research  on  performance  issues  has  occurred  in  many  different  domains.  Where  we  are  today  is 
due  in  part  to  what  has  occurred  in  domains  other  than  ATC.  System  performance  in  air  space  is 
a  function  of  both  controllers  and  pilots. 
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Stein  and  Rosenberg  (1983)  studied  workload  and  pilot  performance  leading  to  new  measurement 
techniques  that  applied  to  both  air  crews  and  air  traffic  controllers.  In  this  study,  pilots  flew 
missions  at  three  levels  of  difficulty,  or  task  load,  which  was  induced  by  turbulence.  Mission 
order  was  counter-balanced  across  pilots.  Pilots  were  asked  to  respond  eveiy  minute  and 
evaluate  their  workload  at  that  time.  They  were  cued  by  a  tone  and  a  light-emitting  diode  on  the 
response  switch  box  mounted  below  the  aircraft's  throttles.  At  this  time,  the  measurement  system 
and  the  theoretical  foundation  behind  it  was  called  the  Pilot  Objective- Subjective  Workload 
Technique  (POSWAT).  The  subjective  scaling  involved  the  responses  already  discussed.  The 
objective  part  was  the  measurement  of  response  delays  and  overall  pilot  performance  on  such 
dimensions  as  flight  technical  error,  which  is  the  degree  to  which  the  pilot  strays  from  his  assigned 
flight  path.  Performance  was  evaluated  against  an  absolute  standard  that  assigned  error  points 
based  on  the  magnitude  of  the  pilots  deviation  from  the  assigned  altitude.  There  was  the  growing 
conception  that  the  measurement  of  workload  is  irrelevant  without  an  evaluation  of  performance. 

The  subjective  real-time  scaling  in  this  study  was  sensitive  to  the  levels  of  taskload.  Workload 
was  related  to  the  segments  of  flight,  being  highest  in  those  segments  that  involve  the  greatest 
demands  on  the  pilot,  takeoff,  final  approach,  and  landing.  Such  confirmation  of  what  has  been 
anecdotal  data  would  not  have  been  possible  using  post-run  scaling. 

Stein  (1984a)  did  a  study  to  determine  whether  there  were  any  measurable  differences  in 
workload  and  performance  between  relatively  new  pilots  and  experienced,  high-time  personnel. 
This  became  known  as  the  masters-joumeyman  study.  Professional  military  and  civilian  pilots,  the 
masters,  were  compared  against  a  unique  group  of  instrument-rated  pilots  who  had  very  low 
experience  levels.  These  journeymen  received  their  pilots'  licenses  through  a  one-time  FAA 
experiment  which  evaluated  the  feasibility  of  instrument  training  for  pilots  with  under  200  hours 
of  experience.  Both  groups  flew  simulated  missions  under  three  levels  of  counter-balanced 
taskload.  Taskload  was  influenced  by  turbulence  and,  at  the  highest  level,  by  the  introduction  of 
an  emergency  condition  toward  the  end  of  the  flight.  The  POSWAT  system  for  the  evaluation  of 
workload  was  used  along  with  the  measurement  capabilities  of  the  flight  simulator  in  order  to 
assess  flight  performance.  Participants  completed  a  post-flight  measure  called  the  Flight 
Workload  Questionnaire.  This  had  four  scales;  workload,  degree  of  busyness,  amount  of  thinking 
required,  and  an  overall  evaluation  of  how  the  pilot  was  feeling. 

Masters  performed  better  in  all  segments  of  flight  than  did  the  journeymen.  Both  inflight 
POSWAT  and  post-flight  ratings  of  workload  showed  higher  workload  for  the  journeymen  than 
for  the  masters.  Correlation  co-efficients  between  workload  and  performance  produced  an 
interesting  phenomenon.  When  all  pilots  were  considered,  the  relationship  was  negative;  higher 
workload  meant  poorer  performance.  However,  this  finding  did  not  appear  within  each  of  the 
subgroups;  it  required  the  full  skill  range  to  appear. 

POSWAT  was  the  beginning  of  real-time  workload  measurement  in  simulation.  It  would  later  be 
modified  to  become  the  Air  Traffic  Workload  Input  Technique  (ATWIT),  which  will  be  described 
in  more  detail.  Murphy  (1987)  performed  what  was  essentially  a  replication  of  the  masters- 
joumeyman  study  under  a  contract  with  the  FAA.  His  results  confirmed  those  found  at  the 
Technical  Center. 

Stein  (1989)  completed  a  study  which  was  designed  to  evaluate  the  impact  of  changing  the 
minimum  legal  separation  permitted  between  aircraft  that  are  approaching  independent  parallel 
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mnways  at  a  major  U.S.  airport.  The  minimum  was  2  nautical  miles  (nmi),  and  the  proposed 
change  was  to  decrease  this  to  1.5  nmi.  Highly-experienced  controller  volunteers  participated  in  a 
simulation  that  involved  controlling  traffic  in  1-hour  sessions,  using  the  two  separation  standards 
alternatively.  Performance  was  measured  with  the  automated  data  collection  capability  of  the 
simulator  and  over-the-shoulder  evaluations.  The  measurement  data  set  was  based  on  earlier 
work  and  will  be  described  more  completely  later  in  this  section.  Workload  was  estimated  every 
half-hour  by  the  evaluators  and  after  every  hour  by  the  participant,  using  a  post-run  questionnaire 
similar  to  that  used  in  the  workload  probe  study  4  years  previously  (Stein,  1985).  ATWIT  was 
not  employed  in  this  study,  however. 

In  terms  of  performance,  controllers  using  the  reduced  separation  did  not  make  any  more  errors 
than  they  did  when  using  the  2  nmi  minimum.  In  fact,  they  were  actually  able  to  land  more  planes 
given  the  reduced  separation.  The  post-run  questionnaire  had  five  separate  scales;  workload, 
performance,  busyness,  stress,  and  workability  of  the  separation  standard.  It  was  tailored  to  the 
specific  experiment.  Overall,  there  were  no  significant  differences  in  perceived  workload  from 
either  the  observers  or  the  participants. 

Buckley,  DeBaryshe,  Hitchner,  and  Kohn  (1983)  performed  two  experiments  to  examine  the  use 
of  simulation  for  the  evaluation  of  air  traffic  controller  performance.  They  emphasized  the  quality 
of  measurement  and  identified  the  basic  dimensions  for  measuring  ATC  functions  in  real  time. 
They  studied  the  issue  of  the  interaction  of  sector  geometry  and  traffic  density  on  various 
performance  measures.  One  outcome  of  the  first  experiment  was  that  there  was  a  statistically 
significant  effect  of  sector  geometry  and  traffic  density  for  almost  all  of  the  10  performance 
measures.  There  was  also  a  significant  interaction  effect  between  geometry  and  density.  The 
authors  suggested  that  "the  nature  and  extent  of  this  interaction  depends  on  the  measures 
involved"  (p.  73). 

This  first  experiment  had  examined  the  effects  on  system  performance  measurements  using  two  en 
route  sector  airspace  layouts  and  three  traffic  density  levels  ranging  from  very  light  to  very  heavy. 
Data  were  collected  from  two  1-hour  runs  for  each  of  3 1  controllers.  Sector  geometry  had  a 
major  impact  on  performance,  and  this  led  them  to  the  design  of  a  second  experiment. 

The  second  experiment  examined  the  effects  of  collecting  a  great  deal  of  data  over  time  by 
repeated  measures.  The  database  was  sufficient  so  that  a  factor  analysis  was  computed  to  look 
for  redundancy  in  the  measures  used  to  quantify  system  performance.  Twelve  1-hour  runs  were 
conducted  using  the  same  sector  with  the  same  traffic  level  for  each  of  39  controllers. 

The  data  resulting  from  Buckley’s  et  al.  (1983)  first  experiment  were  cross-validated  with  the 
factor  analysis  derived  from  the  second  experiment.  This  produced  four  meaningful  factors  or 
measures:  confliction,  occupancy,  communication,  and  delay.  The  confliction  factor  included 
measures  of  3-,  4-,  and  5-mile  conflicts.  The  occupancy  factor  included  measures  of  the  time  an 
aircraft  was  under  control,  distance  flown  under  control,  fuel  consumption  under  control,  and 
time  within  boundary.  The  communication  factor  included  path  changes,  number  of  ground-to  air 
communications,  and  the  duration  of  ground-to-air  communications.  The  delay  factor  included 
total  number  of  delays  and  total  delay  times.  Two  auxiliary  measures,  number  of  aircraft  handled 
and  fuel  consumption,  were  also  relevant.  These  experiments  conducted  by  Buckley  et  al.  have 
served  as  building  blocks  for  most  of  the  controller  performance  research  that  has  followed. 
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Thackray  and  Touchstone  (1988,  1989),  working  at  the  FAA  Civil  Aeromedical  Institute,  also 
were  interested  in  performance.  They  examined  the  performance  implications  of  varied  task 
loads.  They  developed  a  small  scale  ATC  simulation  and  examined  behavior  of  college  students 
required  to  monitor  a  simulated  radar  display  for  two  different  types  of  events.  One  was  a  simple 
task  of  detecting  and  reacting  to  a  change  in  the  altitude  data  block  linked  to  an  aircraft  target. 
The  second  involved  detecting  the  occurrence  of  two  aircraft  on  the  same  flight  path  at  the  same 
altitude,  a  collision  course.  The  authors  saw  this  as  a  more  demanding  task  in  that  the  participant 
had  to  decide  whether  a  conflict  was  imminent  or  not.  Participants  worked  in  2-hour  sessions. 

Thackray  and  Touchstone  (1988,  1989)  did  not  attempt  to  scale  workload;  they  focused  entirely 
on  performance.  They  were  concerned  with  the  frequency  of  correct  detections  and  the  missed 
events.  Results  demonstrated  that  the  low  taskload  events  were  virtually  never  missed  and  that 
response  times  did  not  increase  appreciably  during  the  work  sessions.  For  the  high  taskload 
conflict  detection  task,  events  were  missed  and  the  number  and  latency  increased  over  time.  This 
suggested  fatigue  and/or  some  sort  of  change  in  reserve  capacity  for  information  processing  over 
time.  The  researchers  concluded  that  the  decline  in  performance  with  the  higher  taskload  may 
have  been  based  on  the  amount  of  information  processing  required  and  its  impact  on  the 
employment  of  attentional  resources.  It  was  unfortunate  that  Thackray  and  Touchstone  (1989) 
did  not  use  a  secondary  task  measure  to  more  thoroughly  evaluate  this  hypothesis. 

Another  unfortunate  aspect  of  their  research  was  the  fact  that  they  used  college  students.  Zingale, 
Grromelski,  Ahmed,  and  Stein  (1993)  have  shown  that  college  students  are  not  a  good  model  for 
air  traffic  controllers.  Even  with  considerable  training  in  ATC  concepts  and  using  a  user  friendly 
simulation,  Zingale  et  al.  found  that  college  students  do  not  behave  the  same  way  as  experienced 
controllers,  and  therefore,  results  of  studies  using  students  can  not  be  easily  generalized  to  the 
controller  work  force. 

In  a  study  of  how  actual  controllers  used  their  information  to  facilitate  their  performance.  Means 
et  al.  (1988)  studied  the  way  that  en  route  controllers  organized  aircraft.  They  observed  that 
controllers  recalled  aircraft  in  groups,  invariably  drawing  one  group  at  a  time  when  tested.  When 
asked  to  name  the  groups,  controllers  labeled  them  in  accordance  with  a  specific  type  of  traffic 
issue  (i.e.,  arrivals  or  crossing  traffic  at  a  specific  fix).  Geographical  proximity  played  less  of  a 
role  in  grouping  than  did  the  interaction  and  potential  conflicts  between  members  of  a  group. 

This  takes  training  and  experience.  Organization  of  information  has  been  identified  as  the  one 
factor  which  has  the  greatest  probability  of  improving  cognitive  performance  in  ATC  (Vortac, 
1991). 

Memory,  SA,  and  performance  may  be  related.  There  have  been  a  number  of  definitions  of  SA 
that  have  been  offered  during  its  relatively  short  history.  One  definition  of  SA,suggested  by 
Endsley  (1989)  is:  ”...  the  perception  of  or  the  elements  in  the  environment  within  a  volume  of 
time  and  space,  the  comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the  near 
future.”  Endsley  (1990)  described  the  measurement  model  she  had  created  as  the  Situation 
Awareness  Global  Assessment  Technique  (SAGAT).  SAGAT  involves  developing  a  question  set 
based  on  potential  events  in  a  scenario.  Questions  are  randomly  selected  from  the  set.  The  flight 
scenario  is  frozen  at  a  predetermined  point  in  time.  The  pilot  is  removed  and  asked  to  respond  to 
the  questions.  The  correctness  of  answers  is  determined  by  referring  to  what  was  actually 
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happening  at  the  point  of  scenario  freeze.  Scoring  requires  storing  that  information  in  analog  or 
digital  fashion  so  that  comparisons  can  be  made  post  hoc. 

Endsley  and  Rodgers  (1994)  studied  en  route  ATC  from  the  viewpoint  of  the  requirements 
generated  for  SA.  These  researchers  attempted  to  identify  the  essential  components  of 
information  that  an  en  route  controller  must  have  in  order  to  perform  his  or  her  tasks.  Using  a 
panel  of  eight  SMEs,  the  researchers  employed  a  replay  of  ATC  incidents  to  cue  participant 
memory.  Each  member  of  the  eight-person  panel  was  presented  with  one  or  two  ATC  incidents 
that  were  recreated  on  a  video  screen.  Each  member  was  subsequently  interviewed  about 
information  requirements  for  one  or  more  major  task  areas  in  ATC  (i.e.,  separating  aircraft, 
analyzing  weather  situations).  The  end  product  of  this  work  was  a  series  of  information 
requirements  linked  to  each  aspect  of  the  controllers  duties.  This  may  have  implications  for  future 
ATC  experiments  to  the  extent  that  the  presence  or  absence  of  these  elements  of  information  are 
present  during  the  simulated  ATC  operations.  How  controllers  think  and  use  information  has 
elicited  considerable  interest  and  research. 

This  interest  was  reflected  by  a  family  of  theory  and  research  papers  that  discuss  the  cognitive 
tasks  as  compared  to  the  observable  activities  of  controllers.  It  is  unclear  to  what  extent 
controller  cognition  can  be  effectively  used  to  understand  and  subsequently  measure  performance. 
As  part  of  a  larger  program  aimed  at  improving  controller  training,  a  group  of  researchers 
performed  a  cognitive  task  analysis  of  expertise  to  see  if  experts  and  novices  differed  in  how  they 
think  (Seamster,  Redding,  Cannon,  Ryder,  &  Purcell,  1993).  They  concluded  that  experts  took  a 
wider  view  of  the  evolving  air  traffic  situation.  Experts  appear  to  be  more  flexible  in  their 
approach  to  the  dynamics  in  their  airspace.  The  researchers  identified  13  en  route  controller  tasks 
that  were  linked  to  their  cognitive  models  of  the  airspace;  maintain  situation  awareness,  develop 
and  revise  sector  control  plan,  resolve  aircraft  conflicts,  re-route  aircraft,  manage  arrivals,  manage 
departures,  manage  overflights,  receive  hand-off,  receive  pointouts,  initiate  handoffs,  initiate 
pointouts,  issue  advisories,  and  issue  safety  alerts.  Each  of  these  is  broken  into  numerous 
sub-goals  which  establish  the  matrix  of  the  controller's  mental  model. 

According  Seamster  et  al.  (1993),  their  research  supports  the  hypothesis  that  experienced 
controllers  group  or  organize  their  picture  by  events  rather  than  by  individual  aircraft.  The  mental 
model  and  task  accomplishment  or  requirement  interact  and  influence  each  other.  When  thinking 
out  complex  ATC  problems,  experts  (in  contrast  to  novices)  used  fewer,  but  more  varied, 
planning  strategies  and  had  more  strategies  for  managing  their  workload.  While  the  results  of  this 
research  are  important  for  an  understanding  of  how  controllers  think,  the  researchers  did  not 
attempt  to  relate  their  model  to  actual  controller  performance  measured  objectively. 

There  is  a  diversity  of  opinion  concerning  whether  SA  is  an  indicator  of  performance  itself  or  is 
merely  a  necessary,  but  not  sufficient,  condition  for  adequate  performance.  In  a  memory  study  to 
be  described  later,  the  authors  assumed  that  SA  is  a  precursor  to  performance  and  an  indicator  of 
the  current  level  of  working  memory  (Sollenberger  and  Stein,  1995-In  press).  However,  in  an 
extensive  literature  review  of  workload  and  performance  measurement,  Fischer  (1995-In  press) 
recommends  that  S  A  be  considered  as  a  performance  measure.  She  suggests  that  it  is  the  missing 
link  between  non-observable  controller  cognitive  activity  and  resultant  observable  behaviors.  This 
may  be  an  overstatement  of  the  measurement  power  of  SA.  However,  researchers  will  need  to 
continue  thinking  about  the  role  of  SA  in  the  human  performance  equation. 


9 


This  equation  is  a  complex  matter.  In  an  effort  to  establish  a  starting  point  for  future 
measurement  of  controller  performance,  personnel  at  the  FAA  Technical  Center  developed  a 
compilation  of  measures  that  could  be  used  in  real-time  simulation.  This  effort  is  described  in  the 
following  section. 

1.3.1  Performance  Measurement  Search  For  Dependent  Variables. 

This  current  work  reflects  a  history  of  concern  for  measurement  in  ATC.  There  have  been  a 
number  of  expansive  summaries  of  measurement  tools  for  evaluating  human  performance  and 
workload.  For  example,  a  guide  was  published  (ANSI,  1992)  that  included  measures  such  as 
POSWOT  and  ATWIT.  However,  it  did  not  review  any  of  the  performance  work  and  measures 
in  ATC,  and  only  touched  other  performance  domains,  such  as  piloting  and  even  white  collar 
clerking. 

Researchers  in  ATC  performance  have  been  left  to  their  own  devices  to  establish  the  measures 
that  they  use.  Stein  (1992)  assembled  and  consolidated  the  variables  that  had  been  useful  over  the 
years  for  researchers  at  the  FAA  Technical  Center.  This  work  was  based  primarily  on  the 
research  of  Buckley  et  al.  (1983)  and  to  a  lesser  extent  on  research  accomplished  by  Stein 
(1984a,1984b,  1985).  What  follows  are  excerpts  from  the  unpublished  specification  which  may 
apply  to  this  current  study. 

Simulation  research  has  been  used  to  study  ATC  concepts,  equipment,  and  procedures  for  35 
years  or  more.  Over  this  time  period,  various  sets  of  dependent  variables  have  evolved  to  assist  in 
the  evaluation  of  system  and  individual  controller  performance.  The  specific  subset  of  variables 
has  generally  been  tailored  to  meet  the  research  goals  of  each  study  that  was  run.  Most  of  the 
ATC  studies  using  simulation  have  been  conducted  at  the  Technical  Center. 

It  is  assumed  that  everything  that  occurs  in  the  simulation  is  recordable  and  recoverable  on  a  post- 
hoc  basis.  There  has  been  no  requirement  for  real-time  data  analysis.  All  data  processing  can  be 
accomplished  after  the  completion  of  the  simulation.  It  has  also  been  assumed  that  there  is  a  data 
flow  from  target  generation  through  controller  actions  and  subsequent  results  in  terms  of  aircraft 
responses  that  will  be  recoverable  on  a  post  simulation  basis.  This  implies  that  all  raw  data,  such 
as  the  relative  position  of  aircraft,  are  saved  so  that  further  analyses  can  be  accomplished. 

The  dependent  variables,  described  below,  are  a  subset  of  all  those  that  are  possible.  This  subset 
was  selected  for  its  generality  and  practicality,  as  well  as  for  the  potential  statistical  power  of  the 
measures.  A  researcher  may  wish  to  run  a  full  set  of  analyses  using  all  these  variables  or  some 
less-inclusive  group.  The  interface  for  the  analysis  should  afford  the  opportunity  to  select  those 
variables  desired  for  the  particular  questions  under  study. 

The  majority  of  measures  in  the  dependent  variable  set  are  based  on  frequencies  of  events  and 
time,  both  of  which  should  be  cumulative,  based  on  a  specifiable  time  period.  The  research  design 
may  include  a  hypothesis  of  change  in  conflict  frequencies  and  time  duration  based  on  the  amount 
of  time  that  a  controller  has  been  on  position.  So,  the  ability  to  compute  statistics  based  on  a 
specifiable  time  block  is  important. 

The  variables  apply  to  Terminal  Radar  Approach  Facilities  and  are  presented  with  as  much  detail 
as  necessary  to  facilitate  their  computation.  Unprocessed  variables  are  data  that  should  require  no 
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processing,  but  should  be  available  at  the  end  of  each  simulation  run  directly  from  one  or  more 
storage  files.  They  are  basically  self  explanatory. 

1.3. 1.1  Unprocessed  Variables. 

Aircraft  Identification  and  Flight  Plans 
Run  Number 

Run  Duration(seconds) 

Controller  Identification  Code 
Airspace  Sector  Identification 
Experimental  Condition/Combination 

1.3. 1.2  Conflict  Variables. 

All  conflict  variables  assume  a  technical  violation  of  minimum  separation  between  pairs  of  aircraft 
flying  in  controlled  airspace.  Variable  names  are  arbitrary  and  may  or  may  not  have  been  used  in 
previous  research  efforts.  Variable  concepts,  however,  have  for  the  most  part  been  employed  in 
earlier  work.  Once  a  concept  is  explained,  such  as  the  principle  of  accumulating  time  durations  of 
conflicts,  it  may  or  may  not  be  repeated  in  similar  variable  descriptions. 

a.  SCNFfTERMD  -  Standard  conflicts  in  the  terminal  area.  The  separation  shall  be  3  nmi  or 
1000  feet  of  vertical  separation.  This  variable  will  not  be  useful  on  final  approach  sequences  due 
to  wake  vortex  considerations,  and  longitudinal  separation  violations  will  take  over.  The  analysis 
of  this  variable  will  therefore  have  to  take  into  consideration  the  relative  location  of  aircraft  in  the 
airspace  and  rule  out  those  aircraft  on  final  approach.  This  is  usually  done  by  setting  a  point  in 
space  such  as  the  outer  marker  or  by  defining  the  final  approach  heading  and  filtering  out  any 
conflict  hits  from  aircraft  established  on  final. 

b.  SCNFDfTERMI  -  The  cumulative  duration  of  SC]SlF(iERM)  conflicts. 

c.  XCNFfTERMI  -  The  user  must  have  the  option  of  setting  a  conflict  criteria  that  is  flexible. 
This  becomes  necessary  when  the  purpose  of  the  experiment  is  to  evaluate  the  impact  of  changes 
in  separation  minima  in  the  airspace.  This  measure  applies  primarily  to  aircraft  in  the  Terminal 
Control  Area  (TCA)  which  are  not  on  final  approach. 

d.  LCNF  -  This  is  the  primary  conflict  measure  for  aircraft  that  are  on  final  approaches  and  are 
in  trail  of  one  another.  This  measure  must  take  into  consideration  the  impact  of  wake  vortex 
issues  as  defined  in  71 10.65H  (FAA,  1993).  However,  for  the  purposes  of  research,  some 
assumptions  may  be  made  and  will  be  made  part  of  subsequent  research  designs.  For  example, 
assume  that  3  miles  standard  separation  is  acceptable  unless  wake  vortex  criteria  apply. 

Obviously,  this  measure  will  show  more  conflicts  than,  for  example,  SCNF  and  could  indicate  that 
the  controller  is  not  paying  adequate  attention  to  the  aircraft  types  he/she  is  working. 
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e.  LCNFD  -  The  cumulative  durations  of  LCNF. 


f.  PCNF  -  Parallel  conflict  frequencies.  This  measure  is  used  to  evaluate  conflicts  of  aircraft 
that  are  on  simultaneous  parallel  approaches  to  an  airport.  The  criteria  for  violation  must  be  user 
specifiable  since  the  purpose  of  the  research  may  be  to  examine  the  possible  impact  of  changing 
the  separation  minima. 

g.  PCNFD  -  The  cumulative  durations  of  PCNT. 

h.  BSCNF  -  Between  sector  conflict  frequencies.  Here,  the  sector  boundary  must  be 
identified.  This  is  primarily  a  terminal  measure,  and  a  3-mile  criteria  is  acceptable.  It  is  basically  a 
standard  conflict,  but  the  controller  generally  does  not  have  both  aircraft  unless  he/she  has  taken 
an  early  inbound  handoff. 

i.  BSCNFD  -  The  cumulative  durations  of  BSCNF. 

j.  API  -  Aircraft  proximity  index.  This  is  a  measure  of  conflict  severity  developed  by  Mr.  Lee 
Paul  (1990)  of  the  FAA  Technical  Center.  While  it  can  be  computed  frequently  during  each 
conflict  situation,  it  is  most  useful  at  the  point  of  closest  approach  of  two  aircraft  which  are  in 
violation.  The  computational  procedures  are  appended  to  this  specification.  The  API  for  each 
conflict  situation  should  be  recoverable  at  the  end  of  the  simulation,  along  with  a  mean  API  score 
for  the  entire  simulation  and  sub-blocks  of  the  total  time  period.  For  every  API  computed  the 
software  should  provide  output  on  the  actual  horizontal  and  vertical  separation  at  the  point  of 
closest  approach  between  the  aircraft  pair. 

k.  AS  CNF  -  This  is  the  frequency  with  which  the  controller  allows  aircraft  to  conflict  with 
restricted  airspace.  The  boundaries  of  the  airspace  in  question  shall  be  user  specifiable.  An 
airspace  conflict  occurs  when  an  aircraft  actually  crosses  a  boundary. 

l.  ASCNFD  -  The  cumulative  durations  of  ASCNF. 

The  nature  and  number  of  conflict  measures  does  not  imply  that  controllers  are  not  effective  or 
that  they  are  unsafe.  However,  previous  research  has  shown  that  such  measures  can  be  effective 
especially  when  trying  to  evaluate  the  effects  of  change  on  the  person-machine  system. 

1.3. 1,3  Complexity  Measures. 

The  concept  of  complexity  means  different  things  to  ATC  personnel.  There  is  no  one  generally 
agreed-upon  definition.  Most  controllers  comment  on  the  aircraft  frequency  and/or  the  number  of 
control  actions  they  have  to  take  with  each  under  their  control  until  they  can  hand  it  off. 

a.  CMAV  -  This  is  a  very  basic  measure  of  what  might  better  be  called  system  activity.  It  is 
simply  the  average  number  of  aircraft  within  X  miles  of  each  other  with  X  being  user-specifiable. 

A  sampling  rate  on  this  of  once  every  10  seconds  would  be  adequate. 

b.  ALT  -  The  frequency  of  altitude  change  messages  sent  from  the  controller  to  the  aircraft. 
This  is  cumulated  over  the  run  and/or  the  prespecified  time  block  within  the  run. 
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c.  HDG  -  The  frequency  of  heading  change  messages  sent  from  the  controller  to  the  aircraft. 

d.  SPEED  -  The  frequency  of  speed  change  messages  sent  from  the  controller  to  the  aircraft. 

1.3. 1.4  Non  Conflict  Errors. 

a.  MISSAPP  -  The  frequency  of  missed  approaches  in  the  terminal  control  area.  This  should 
include  both  the  primary  airport  and  any  satellite  airports  to  which  approaches  are  being 
controlled.  This  is  cumulated  over  the  run  or  the  pre-specified  time  block. 

b.  HOFFMISS  -  The  frequency  with  which  the  controller  allows  aircraft  to  leave  his/her  sector 
without  a  formal  handoff  to  the  receiving  sector.  This  measure  can  be  used  in  both  the  terminal 
and  en  route  environments. 

c.  HOFFERR  -  The  frequency  with  which  the  controller  attempted  to  handoff  an  aircraft  at  the 
incorrect  altitude  or  airspeed.  These  errors  shall  be  cumulated  over  the  entire  run  or  based  on 
pre-specified  time  blocks. 

d.  NDLY  -  The  frequency  of  hold  messages  sent  to  aircraft  and  the  number  of  turns  of  greater 
than  100  seconds  duration.  These  two  elements  are  summed  and  cumulated  for  the  duration  of 
the  run  or  based  on  pre-specified  time  blocks. 

e.  COMPLY  -  An  accumulated  time  variable  based  on  the  durations  of  time  between  the 
aircraft  calls  for  service  and  the  controller’s  initial  response.  The  time,  in  seconds,  is  cumulated 
during  the  entire  run  or  based  on  pre-specified  time  blocks. 

f  COMDLYNBR  -  The  cumulated  frequency  of  comdlys  that  exceed  20  seconds.  This  is 
based  on  the  whole  run  or  pre-specified  time  blocks. 

1 .3 . 1 . 5  Communication  Activity. 

a.  VOIFREO  -  The  frequency  of  voice  communication  from  the  controller  to  the  aircraft  under 
control  if  voice  is  used  in  the  simulation.  Each  time  the  microphone  is  keyed  shall  be  counted  as 
one  transmission.  These  shall  be  cumulated  over  the  run  or  based  on  pre-specified  time  blocks. 

b.  VOIDUR  -  the  cumulated  time  in  seconds  that  the  controller  is  transmitting  to  aircraft 
under  control.  This  is  based  on  the  whole  run  or  pre-specified  time  blocks. 

c.  CKEY  -  The  accumulated  total  frequency  of  controller  keystrokes  on  the  keypad.  This  is 
based  on  the  whole  run  or  pre-specified  time  blocks. 

d.  PKEY  -  The  accumulated  frequency  of  simulation  pilot  keystrokes  if  simulation  pitots  are 
part  of  the  simulation.  This  is  based  on  the  whole  run  or  pre-specified  time  blocks. 

1.3. 1.6  Activitv/Taskload. 

a.  NFLT  -  The  number  of  flights  handled  by  the  controller  for  a  given  period  of  time  based  on 
the  entire  run  or  on  pre-specified  time  blocks. 
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b.  LAND-  The  number  of  landings  completed  by  a  controller  working  final  approach  in  a 
TRACON  environment.  This  is  based  on  the  entire  run  or  pre-specified  time  blocks. 

c.  DEPART  -  The  number  of  departures  worked  by  a  controller  working  departure  control  in 
a  TRACON  environment.  This  is  based  on  the  entire  run  or  pre-specified  time  blocks. 

d.  HANDOFF  -  The  number  of  successful  handoffs  to  adjacent  sectors  or  facilities.  This 
applies  to  both  en  route  and  TRACON  environments. 

e.  ATWIT  -  The  Air  Traffic  Workload  Input  Technique.  This  measure  was  first  used  by  Stein 
(1985).  It  grew  out  of  earlier  research  with  pilots  (Rosenberg,  Rehmann  &  Stein,  1982).  The 
software  cues  the  operator  that  a  response  is  required.  This  occurs  on  a  pre-set  time  frame,  such 
as  once  per  minute.  The  controller  pushes  a  button  numbered  fi-om  1  to  10  and  the  system 
records  the  button  push  and  response  latency.  The  final  product  of  ATWIT  is  an  average  of  the 
scaled  responses  for  the  whole  run  or  for  user-specified  time  blocks. 

This  concludes  the  summary  of  the  simulation  performance  variables  that  are  currently  in  use.  It 
does  not,  however,  limit  researchers  to  only  these  variables.  The  future  will  determine  the  variety 
of  measurement  tools  that  researchers  are  able  to  create. 

The  simulation  measurement  described  above  has  been  recently  used  in  the  FAA  Technical  Center 
Human  Factors  Laboratory.  Sollenberger  and  Stein  (1995,  In  press)  conducted  a  study  of 
controller  memory  issues  to  determine  whether  performance  could  be  enhanced  using  a  memory 
aid. 

This  experiment  used  a  training  simulation  called  ATCoach,  which  was  adapted  for  research.  The 
performance  measures  were  collected  automatically  when  each  of  16  controllers  worked  the 
simulation.  The  participating  controllers  worked  traffic  using  their  own  airspace  from  Atlantic 
City  International  Airport  (ACY).  They  worked  traffic  under  high  and  low  task  load  and  under 
normal  and  memory-aided  conditions.  The  memory  aides  were  based  on  increased  structure  that 
involved  pitots  doing  more  of  the  navigation,  using  preplanned  routes,  and  requiring  controllers  to 
do  less,  providing,  in  theory,  more  time  for  the  controllers  to  engage  in  memory-enhancing 
activities  such  as  maintenance  rehearsal  and  note  taking  and/or  strip  marking. 

The  performance  measures  cited  earlier  were  analyzed  to  determine  if  memory  aiding 
accomplished  anything.  Unfortunately,  the  memory  aides  did  not  seem  to  improve  controllers’ 

SA  as  it  was  measured.  They  also  did  not  improve  the  controllers’  memory  for  aircraft 
information,  such  as  the  last  commands  he  or  she  gave.  On  the  positive  side,  the  memory  aides 
did  have  some  positive  influence  on  controllers’  behavior,  as  recorded  in  the  automated 
performance  measurement  data.  In  the  aided  condition,  controllers  made  significantly  fewer 
ground  to  air  transmissions.  Also,  they  gave  fewer  altitude  and  heading  changes.  These  variables 
have  been  used  as  indicators  of  controller  workload.  Another  positive  result  was  that,  with  the 
memory  aides,  controllers  made  fewer  handoff  errors.  Without  the  performance  indicators,  these 
positive  findings  would  not  have  been  discovered. 
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2  EXPERIMENT. 


2.1  PURPOSE. 

The  purpose  of  this  research  was  to  develop  and  validate  a  generic  sector  that  could  be  used  to 
evaluate  air  traffic  controller  performance  in  a  standard  fashion.  The  study  had  two  goals.  The 
first  was  to  evaluate  controllers’  ability  to  learn  a  new  sector  in  a  short  amount  of  time.  The 
second  was  to  evaluate  how  similar  or  different  controller  and  system  performance  would  be  on  a 
generic  versus  a  home  sector,  in  this  case  the  ACY  sector. 

2.2  LOGIC  BEHIND  A  GENERIC  SECTOR. 

A  new  program  titled  the  Separation  and  Control  Hiring  Assessment  (SACHA)  is  underway  to 
develop  valid  measures  for  selecting  ATC  specialists  in  the  FAA.  The  SACHA  program  has  the 
goal  of  developing  a  valid  selection  process  (i.e.,  test  battery)  for  selecting  air  traffic  controllers. 
Component  tasks  of  this  project  include  job  analysis,  predictor  development,  criteria  development, 
and  validation  of  these  predictors  and  criteria.  Ideally,  these  performance  criteria  would  be  based 
on  measures  taken  from  the  controller's  own  sector.  However,  a  controller's  performance  may 
vary  depending  on  the  amount  of  time  he  or  she  has  been  working  the  sector.  In  addition,  sectors 
vary  in  complexity  and,  therefore,  in  difficulty  for  the  controller.  A  standard  generic  sector  could 
be  a  potential  solution  in  that  all  the  conditions  under  which  performance  is  measured  would  be 
the  same  for  all  participants.  This  would  be  a  significant  advantage  over  using  performance 
measured  on  each  controller's  home  sector  where  many  factors,  such  as  familiarity  and  sector 
complexity,  vary. 

In  order  to  perform  the  study,  a  generic  sector  had  to  first  be  defined  and  developed.  In  the 
context  of  this  research,  generic  refers  to  a  sector  which  embodies  the  important  elements  of  a 
terminal  sector  (i.e.,  arrival  and  departure  routes,  terminal  radar  range  and  performance,  and  radar 
procedures).  In  order  to  achieve  the  goals  of  the  study,  the  test  generic  sector  was  specifically 
designed  to  be  quite  different  from  the  home  sector.  The  reasoning  behind  making  the  generic 
sector  different  is  that  this  would  require  learning  on  the  part  of  the  controller,  and  the  controller's 
performance  on  this  sector  would  be  based  on  his  or  her  skill  as  a  controller  and  his  or  her 
mastery  of  the  sector.  In  addition,  it  is  likely  that  participants  in  the  SACHA  testing  would  come 
from  all  parts  of  the  country  and  therefore  from  diverse  facilities  and  operations.  Creating  a 
sector  which  is  different  from  the  controller's  home  sector  would  allow  us  to  pilot  test  a  situation 
that  would  be  very  likely  to  occur  during  SACHA  testing.  Specific  items  of  differentiation  include 
the  route  structure,  the  mixture  of  traffic,  the  letters  of  agreement  (LOAs)  between  the  sector  and 
adjacent  facilities,  the  names  of  fixes,  the  direction  of  traffic  flow,  and  the  placement  and 
orientation  of  sector  boundaries. 

2.3  AIRSPACE  AND  TRAFFIC  SCENARIOS. 

2.3.1  Generic  Sector  Airspace  and  Scenarios. 

One  of  the  primary  concerns  of  this  effort  was  that  the  generic  airspace  appears  realistic  to  a  FPL 
controller,  yet  could  be  learned  with  a  minimal  amount  of  training.  To  achieve  this  objective,  an 
ATC  specialist  was  heavily  involved  in  the  development  of  the  airspace  and  traffic  scenarios.  The 
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specialist  had  extensive  experience  working  in  a  Level  V  terminal  facility  and  had  visited  many 
other  major  terminal  facilities  across  the  United  States  as  a  part  of  previous  projects. 

The  generic  airspace  evaluated  in  this  study  was  based  on  a  four-comer  post  operation  typically 
used  in  many  terminal  areas  in  the  United  States.  Arrival  aircraft  originated  from  one  of  four 
arrival  fixes  just  outside  the  sector  boundaries.  The  arrival  aircraft  traveled  down  corridors  which 
converged  at  an  arrival  transition  fix  near  the  main  airport.  These  arrival  routes  can  be  thought  of 
as  spokes  of  a  wheel  with  the  main  airport  site  as  the  hub.  In  addition  to  the  main  airport,  there 
were  three  satellite  airports  that  were  under  radar  control.  Departure  aircraft  from  the  main  and 
satellite  airports  were  sent  directly  to  one  of  four  departure  fixes  located  outside  the  sector 
boundaries. 

To  expedite  learning  of  these  fixes,  the  five-letter  identifiers  for  these  intersections  corresponded 
to  their  magnetic  heading  location  relative  to  the  radar  antenna.  For  example,  the  northwest 
arrival  fix  was  named  "NOWES”  and  the  northeast  arrival  fix  was  named  "NEAST."  Departure 
fixes  were  also  given  names  corresponding  to  their  magnetic  headings,  but  all  fix  names  ended  in  a 
"D"  to  denote  a  departure  fix.  For  example,  the  west  departure  fix  five-letter  identifier  was 
"WESTD",  the  east  departure  fix  was  named  "EASTD",  and  the  southwest  departure  fix  was 
named  "SWEDD."  Another  significant  feature  of  the  generic  sector  was  the  use  of  structured 
corridors  to  the  en  route  airspace  for  both  arrival  and  departure  aircraft.  These  corridors  were 
used  to  hand  off  aircraft  to  the  center,  accept  handoffs  from  the  center,  and  provide  path  ways  to 
and  from  the  main  airport.  A  map  of  the  generic  sector  is  presented  in  figure  1 . 

The  generic  sector  also  employed  stmctured  altitude  shelves  which  provided  safe  altitude 
separation  between  aircraft  when  arrival  and  departure  airspace  overlapped.  Arrival  aircraft 
originating  from  the  northeast  and  northwest  had  to  maintain  at  or  above  8,000  feet  until  they 
were  within  a  15-mile  radius  of  the  main  airport.  Departure  aircraft  destined  for  the  east  and  west 
departure  fixes  had  to  maintain  7,000  feet  or  below  until  they  were  15  miles  east  or  west  of  the 
main  airport.  The  altitude  shelves  were  designed  to  add  some  complexity  to  the  sector  in  that  the 
controllers  had  to  differentiate  between  north  departures,  which  could  ascend  immediately,  and 
the  east/west  departures.  Figures  2  and  3  illustrate  the  altitude  shelves  and  their  boundaries. 

LOAs  were  made  up  to  provide  the  controllers  with  standardized  hand-off  procedures.  Two  en 
route  centers  were  created,  one  for  transitioning  aircraft  in  the  northern  portion  of  the  sector 
(North  Central  Center)  and  one  for  transitioning  aircraft  in  the  southern  portion  of  the  sector 
(South  Central  Center).  In  addition,  there  were  four  airports  adjacent  to  the  generic  sector  which 
provided  arrival  traffic  and  accepted  departure  traffic.  LOAs  were  written  to  define  operating 
procedures  between  each  of  these  adjacent  airports  and  the  main  airport.  Copies  of  these  LOAs 
are  found  in  appendix  A  and  B. 

The  traffic  mixture  for  the  generic  sector  was  based  on  actual  flights  from  the  Official  Airline 
Guide  (OAG)  1993.  This  guide  contains  flight  origination  and  destination  information,  as  well  as 
call  signs  and  aircraft  types.  Flights  were  taken  from  this  guide  and  a  database  was  formed  from 
flights  arriving  into  major  metropolitan  terminal  areas.  Approximately  70  percent  of  the  flights 
into  and  out  of  the  generic  sector  were  composed  of  transport  aircraft  including  heavy  aircraft 
(i.e.,  DC-10,  LlOll,  and  747)  and  medium-performance  aircraft  (i.e.,  727,  DC-9,  and  MD-80). 
The  remaining  30  percent  of  the  mixture  were  general  aviation  aircraft,  including  commuter  jets 
(i.e.,  Learjet,  Cessna  Citation)  and  both  single-  and  twin-  engine  propeller-driven  aircraft  (i.e.. 
Piper  Cherokee,  DeHavailland  Dash  6). 
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Scenarios  were  constructed  that  accurately  simulated  traffic  into  a  major  metropolitan  terminal 
area.  Most  of  the  aircraft  were  arriving  to  and  departing  from  the  major  airport  (Genera 
International  Airport)  which  incorporated  a  set  of  parallel  runways.  However,  a  proportion  of  the 
aircraft  arrived  and  departed  from  the  three  satellite  airports  in  the  sector.  A  north  operation  was 
used  for  all  generic  sector  scenarios  in  which  all  arriving  aircraft  used  runway  36R  and  departing 
aircraft  used  runway  36L.  The  satellite  airports  in  the  sector  all  used  a  north  operation  as  well. 

All  scenarios  started  with  a  build-up  of  traffic  during  the  first  5  minutes.  An  SME  controlled  this 
traffic.  At  5  minutes,  the  SME  gave  the  participant  controller  a  relief  briefing  and  then  gave 
control  over  to  him  or  her.  Typically,  there  were  at  least  five  aircraft  on  the  scope  at  this  point. 
Aircraft  steadily  appeared  until  the  conclusion  of  the  scenario.  The  scheduled  rate  of  appearance 
for  aircraft  was  changed  to  represent  either  low,  moderate  or  extremely  busy  traffic  conditions. 

2.3.2  ACY  Airspace  and  Scenarios. 

One  of  the  primary  concerns  in  this  experiment  was  to  create  a  realistic  simulation  of  the  ACY 
airspace  for  the  controllers.  Fortunately,  during  previous  simulations,  a  large  amount  of  data  had 
been  gathered  on  ACY  operations,  normal  operating  procedures,  as  well  as  airspace  boundary 
data  and  LOAs  between  ACY  sector  and  adjacent  facilities.  This  data  was  used  to  create  a 
realistic  depiction  of  the  ACY  airspace  and  construct  realistic  traffic  scenarios.  It  was  believed 
that  the  efforts  invested  in  creating  a  realistic  simulation  of  the  home  sector  would  motivate 
participant  controllers  and  increase  the  credibility  of  the  research  results.  Using  the  information 
obtained  from  the  ACY  tower,  the  airspace  was  constructed  with  a  few  minor  deviations  from  the 
radar  map  used  in  actual  operations.  Of  the  six  airports  in  the  simulated  airspace  vicinity,  only 
four  were  represented  because  two  typically  have  very  little  traffic.  The  ACY  airspace  is 
illustrated  in  figure  4. 

The  traffic  mixture  for  the  ACY  airspace  was  based  on  information  contained  in  actual  flight 
strips  gathered  at  the  tower.  Information  on  the  strips  contained  aircraft  call  signs,  aircraft  types, 
and  flight  plan  information,  and  databases  were  formed  based  on  that  information.  The  majority 
of  the  flights  into  and  out  of  ACY  sector  are  general  aviation  aircraft  and  the  air  carriers  that  do 
fly  into  ACY  are  twin  engine  commuter  aircraft  or  small  transports.  No  heavy  aircraft  were 
included  in  the  traffic  mixture  since  there  were  none  represented  in  the  flight  strip  database. 

Scenarios  were  constructed  that  accurately  simulated  ACY  air  traffic  patterns.  Many  of  the 
aircraft  call  signs  were  familiar  to  controllers  and  represented  common  air  carriers  that  operate  in 
ACY  airspace.  Most  of  the  aircraft  were  arriving  to,  or  departing  from,  ACY.  However,  a  small 
proportion  of  the  aircraft  used  the  three  satellite  airports  in  the  sector.  A  southeast  operation  was 
used  for  all  ACY  scenarios  in  that  all  aircraft  landed  and  departed  from  runway  13.  Aircraft 
arriving  and  departing  from  the  satellite  airports  also  employed  a  southeast  operation.  All 
scenarios  started  with  a  build-up  of  traffic  during  the  first  5  minutes  controlled  by  an  SME.  At  5 
minutes,  the  SME  gave  the  participant  controller  a  relief  briefing  and  then  gave  control  over  to 
the  participant  controller.  Typically,  there  were  at  least  five  aircraft  on  the  scope  at  this  point, 
with  more  steadily  appearing  until  the  conclusion  of  the  scenario.  The  scheduled  rate  of 
appearance  for  aircraft  was  changed  to  represent  either  low  traffic  conditions  or  extremely  busy 
traffic  conditions. 
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3.  METHOD. 


3.1  PARTICIPANTS. 

Eleven  air  traffic  controllers  from  ACY  Tower  volunteered  for  this  study  and  were  assured  of 
their  anonymity  and  confidentiality.  All  participants  were  FPL  controllers  with  normal  or 
corrected-to-normal  vision  and  had  actively  controlled  traffic  for  the  12  months  prior  to  the  study. 
A  demographic  form  was  completed  by  each  controller  to  describe  the  background 
characteristics  of  the  participants  in  this  study.  Controllers  ranged  in  age  from  32  to  41  years  old 
(Mean=36.7,  SD=3.13)  and  ranged  in  experience  from  4  to  22  years  of  active  service 
(Mean=l  1.3,  SD=5.5).  Additionally,  controllers  provided  self  ratings  of  four  personal  attributes 

that  could  affect  simulation  performance.  Ratings  were  indicated  on  a  scale  ranging  fi'om  1 
(low/poor)  to  10  (high/good)  on  each  question.  The  attributes  included  skill  (Mean=8.2, 
SD-0.92),  motivation  (Mean-7.8,  SD=1.81),  and  health  (Mean=9,  SD=0.94).  The  last  attribute 
was  video  game  experience,  which  was  measured  in  terms  of  hours  per  month  (Mean=2.7, 
SD=3.4).  The  purpose  of  the  last  question  was  based  on  a  finding  with  low  fidelity  simulation 
that  video  game  experience  could  have  an  impact  on  controller  performance.  However,  such  an 
effect  was  not  anticipated  in  this  high  fidelity  simulation  study. 

3.2  SIMULATION  FACILITY. 

The  experiment  was  conducted  in  the  Human  Factors  Laboratory  (HFL)  at  the  FAA  Technical 
Center  in  Atlantic  City,  New  Jersey.  The  experimental  apparatus  consisted  of  a  state-of-the-art 
controller  work  station  with  a  high  resolution  graphics  display,  voice  communication  equipment, 
networked  computer  resources,  and  ATCoach  simulation  software  (copyright  UFA  INC.,  1992). 
The  simulation  was  conducted  by  a  research  psychologist  and  an  ATC  specialist  who  observed  the 
participant  in  the  experiment  room.  A  voice  communication  link  to  another  experiment  room 
allowed  the  controller  to  issue  commands  to  personnel  serving  as  simulation  pilots.  Two 
simulation  pilots  provided  realistic  voice  feedback  to  the  controller  and  controlled  the  movement 
of  the  radar  targets  using  simple  keyboard  commands.  Additionally,  the  simulation  pilots  served 
as  ghost  controllers  to  simulate  coordination  with  controllers  in  charge  of  the  release  of  aircraft  in 
Instrument  Flight  Rules  (EFR)  airspace.  As  part  of  the  simulation,  flight  progress  strips  were 
printed  and  time-ordered  in  a  strip  bay  prior  to  the  start  of  each  scenario.  Figure  5  provides  an 
example  of  flight  strips  in  a  time-ordered  sequence.  The  bold  face  data  block  represents  the 
arrival  time  in  minutes  into  the  scenario.  During  the  simulation,  audio-visual  equipment  was  used 
to  video  record  the  participant's  activities.  Video  tapes  were  made  of  the  radar  display  and  of  the 
controller  as  he  or  she  controlled  traffic  during  the  simulation.  The  audio  from  the  simulation, 
which  included  the  controller  and  pseudo-pilot  communications,  was  also  recorded  on  the  video 
tape. 

3.3  EXPERIMENTAL  DESIGN. 

A  quasi-experimental  approach  was  the  methodology  used  to  answer  the  experimental  questions. 
Quasi-experimental  designs  are  often  used  in  field  research  or  a  field  setting  where  treatments 
differ  on  a  number  of  variables  and  experimental  control  of  a  single  variable  is  not  possible  (Gay, 
1994).  Such  is  the  case  when  comparing  or  correlating  performance  on  sectors  where  many 
factors  are  different. 
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FIGURE  5.  ILLUSTRATION  OF  FLIGHT  STRIPS  ARRANGED 
IN  A  TIME— ORDERED  SEQUENCE 

The  experimental  design  is  illustrated  in  table  1.  It  followed  a  time-series  approach  where  a 
number  of  treatments  are  ordered  chronologically  and  measurements  are  taken  after  each 
treatment.  Each  controller  participated  in  9  scenarios  divided  between  2  days.  The  first  day  was 
considered  a  training  day  where  the  participant  controlled  traffic  on  a  low  volume  ACY  scenario, 
and  then  controlled  traffic  on  four  medium  traffic  volume  generic  sector  scenarios.  The  low 
volume  scenario  consisted  of  30  aircraft  in  a  60-minute  time  period.  This  corresponded  to  a  rate 
of  approximately  7  aircraft  entering  the  scenario  every  15  minutes. 
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TABLE  1.  A  SUMMARY  OF  THE  EXPERIMENTAL  DESIGN. 


DAY 

SCENARIOS  (traffic  volume) 

First 

1  ACY  (low) 

4  Generic  (medium) 

Second 

2  ACY  (low,  high) 

2  Generic  (low,  high) 

Each  of  the  4  generic-sector  runs  lasted  45  minutes.  Each  scenario  consisted  of  1 5  arrival  and  1 5 
departure  aircraft.  This  corresponded  approximately  to  a  rate  of  10  aircraft  entering  the  scenario 
every  15  minutes.  In  addition,  the  four  generic  scenarios  were  matched  in  terms  of  the  entry  times 
for  aircraft  into  the  scenario.  This  was  done  to  attempt  to  balance  the  flow  of  traffic.  However, 
for  a  given  time  slot,  the  aircraft  could  be  a  departure  or  an  arrival.  In  addition,  there  was  no 
attempt  to  systematically  order  the  flight  plans  in  these  scenarios,  so  the  traffic  pattern  in  one 
scenario  was  not  predictable  from  the  traffic  patterns  in  preceding  scenarios.  These  four  generic- 
sector  runs  were  counter-balanced  to  evenly  distribute  any  differences  in  difficulty  that  might  exist 
between  the  four  moderate  level  scenarios. 

The  four  remaining  runs  were  completed  on  a  second-day  session.  This  was  considered  a  test  day 
since,  by  this  point,  the  participant  had  received  a  full  day  of  hands-on  training  on  the  generic 
sector.  Each  controller  worked  two  ACY  scenarios  in  the  morning,  first  a  60-  minute,  low-traffic 
volume  scenario  and  then  a  60-minute  high-traffic  volume  scenario.  Each  high  volume  scenario 
consisted  of  22  departure  and  22  arrival  aircraft.  This  corresponded  to  a  rate  of  1 1  aircraft 
entering  the  scenario  every  15  minutes. 

Each  controller  worked  two  generic  sector  scenarios  in  the  afternoon,  first  a  60-minute,  low- 
volume  scenario,  and  then  a  60-minute  high-volume  scenario.  These  scenarios  were  matched  with 
the  ACY  scenarios  in  terms  of  the  number  of  arrivals  and  departures,  as  well  as  entry  times  of 
aircraft  into  the  scenario.  However,  the  aircraft  destinations  and  flight  plans  were  not 
systematically  ordered,  so  the  traffic  patterns  were  not  predictable  from  working  the  previous 
scenario. 

The  main  dependent  variables  of  the  experiment  can  be  categorized  into  four  distinct  categories  of 
interest.  The  first  category  was  system  effectiveness.  The  present  experiment  used  a  long  list  of 
ATC  performance  measures  that  have  been  examined  in  previous  research  (Buckley,  et  al.  1983; 
Stein  &  Buckley,  1990).  The  present  study  focused  on  10  of  these  system  effectiveness  variables. 
They  included  the  number  of  conflicts,  clustering  of  aircraft,  number  of  communications,  number 
of  clearances,  and  total  distance  the  aircraft  flew  in  the  scenario.  The  second  category  was 
controller  workload  which  was  assessed  using  ATWIT  and  through  items  on  a  post-scenario 
questionnaire.  A  third  category  was  controller  performance  as  measured  by  an  over-the-shoulder 
observer  using  an  over-the-shoulder  rating  form  as  a  measurement  tool.  This  form  is  depicted  as 
appendix  C.  It  incorporated  behavioral-anchored  rating  scales  which  provided  behavioral 
examples  of  poor,  medium,  and  outstanding  performance  on  a  number  of  air  traffic  controller 
performance  dimensions.  These  examples  correspond,  or  are  anchored,  to  scales  points  on  a 
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Likert-type  scale,  Eight  dimensions  of  the  form  were  used  for  the  over-the-shoulder  rating  and 
included  the  following  areas:  communication,  adaptability  and  flexibility,  managing  multiple  tasks, 
and  maintaining  a  safe  and  efficient  traffic  flow.  The  last  area  of  interest  was  the  controller's  self- 
assessment  of  his  or  her  own  performance.  This  was  measured  by  a  post-scenario  questionnaire 
administered  immediately  after  the  controller  finished  the  scenario.  The  self  report  ratings 
reflected  categories  used  currently  in  the  terminal  and  en  route  centers  for  performance  rating. 
Dimensions  included  communication,  prioritization,  pre-planning,  stress,  and  safety.  In  addition, 
an  item  regarding  the  degree  to  which  the  controller  thought  he  or  she  could  improve  with 
practice  was  added  to  examine  the  controller's  self-assessment  of  mastery  on  the  generic  sector. 
The  entire  list  of  dependent  measures  is  presented  in  table  2. 

3  4  PROCEDURE. 

A  training  program  was  developed  to  assist  controllers  in  learning  the  generic  sector  and  the 
procedures  associated  with  controlling  arrival  and  departure  traffic.  A  manual  detailing  the 
operating  procedures  and  LOAs  associated  with  the  generic  sector  was  developed.  This  manual 
contained  detailed  maps  of  the  sector  layout,  as  well  as  altitude  restrictions  for  the  arrival  and 
departure  airspace,  and  was  provided  to  the  participants  before  they  arrived  for  their  first  day 
session.  A  copy  of  the  manual  is  in  appendix  D. 

When  controllers  arrived  at  the  HFL,  they  were  briefed  as  to  how  the  experiment  was  going  to  be 
conducted,  what  was  expected  from  them,  and  their  rights  as  volunteers.  At  this  point,  each 
controller  was  asked  for  their  verbal  informed  consent  to  participate  in  the  study.  Next, 
controllers  completed  a  demographic  form  which  asked  them  about  their  age  and  experience  as  an 
air  traffic  controller,  as  well  as  other  variables  that  might  affect  their  performance  in  the 
simulation. 

On  the  first-day  session,  each  controller  first  worked  a  one-hour  ACY,  low  traffic  volume 
scenario.  After  this  scenario  was  completed,  an  ATC  specialist  briefed  each  controller  on  the 
generic  sector.  This  briefing  included  text  presentations  and  visual  aids,  including  a  static 
presentation  of  the  generic  sector  on  the  radar  screen.  The  specialist  reviewed  the  LOAs,  the  fix 
names,  locations,  and  gate  altitudes  for  the  instrument  landing  system.  Each  controller  was  given 
a  chance  to  ask  any  questions  and  then  worked  the  first  generic  scenario.  As  controllers  worked 
each  scenario,  an  ATC  specialist  made  over-the-shoulder  observations  and  completed  the  rating 
form.  After  each  scenario  was  finished,  controllers  completed  a  self-assessment  of  their  own 
performance  in  a  Post-Scenario  Questionnaire.  At  the  conclusion  of  the  final  day  of  testing, 
participants  were  asked  to  fill  out  a  final  questionnaire  and  were  given  an  opportunity  to  comment 
on  their  experiences. 

The  presentation  order  of  scenarios  and  counterbalancing  features  of  the  experimental  design  are 
illustrated  in  table  3.  All  participants  completed  an  ACY  low  scenario  and  four  generic  medium 
scenarios  on  their  first  day  of  testing.  The  ordering  for  the  four  generic  medium  scenarios  was 
counterbalanced.  On  the  second  day  of  testing,  all  participants  completed  2  ACY  scenarios  (low, 
high)  and  a  20-minute  warm-up  on  the  generic  sector.  In  the  afternoon,  all  participants  completed 
2  generic  sector  scenarios  (low,  high).  Each  controller  completed  each  scenario  only  once. 
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TABLE  2.  THE  LIST  OF  DEPENDENT  VARIABLES. 


System  Effectiveness  Variables 

Cumulative  Average  of  System  Activity  or  Aircraft  Density 

(Number  of  aircraft  with  10  miles  of  another  aircraft) 

Number  of  Altitude  Assignments 

Number  of  Heading  Assignments 

Number  of  Speed  Assignments 

Number  of  Communications 

Duration  of  Communications 

Pseudo-Pilot  Keystrokes 

Total  Time  Under  Control 

Total  Distance  Flown 

Average  Arrival  Interval 

Controller  Workload  Variables 


Air  Traffic  Workload  Input  Technique  Rating 
Over-the-Shoulder  Ratings 


Communicating  and  Informing 

Managing  Multiple  Tasks 

Technical  Knowledge 

Reacting  to  Stress 

Maintaining  Attention  and  Vigilance 

Prioritizing 

Maintaining  a  Safe  and  Efficient  Traffic  Flow 
Adaptability  and  Flexibility 

Self-Assessment  Rating  Categories  ('Post-Scenario  Questionnaire) 
Workload 

Information  Exchange 

Attention 

Prioritization 

Technical  Knowledge 

Phraseology 

Pre-planning 

Traffic  Flow 

Stress 

Safety 
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TABLE  3.  THE  PRESENTATION  ORDER  OF  SCENARIOS  AND  COUNTERBALANCING 
FEATURES  OF  THE  EXPERIMENTAL  DESIGN. 


Participant 

Day  1 

Day  2 

1 

ALl 

GMl 

GM2 

GM3 

GM4 

AL2 

AHl 

GLl 

GHl 

2 

ALl 

GM2 

GM3 

GM4 

GMl 

AL2 

AHl 

GLl 

GHl 

3 

ALl 

GM3 

GM4 

GMl 

GM2 

AL2 

AHl 

GLl 

GHl 

4 

ALl 

GM4 

GMl 

GM2 

GM3 

AL2 

AHl 

GLl 

GHl 

5 

ALl 

GMl 

GM2 

GM3 

GM4 

AL2 

AHl 

GLl 

GHl 

6 

ALl 

GM2 

GM3 

GM4 

GMl 

AL2 

AHl 

GLl 

GHl 

7 

ALl 

GM3 

GM4 

GMl 

GM2 

AL2 

AHl 

GLl 

GHl 

8 

ALl 

GM4 

GMl 

GM2 

GM3 

AL2 

AHl 

GLl 

GHl 

9 

ALl 

GMl 

GM2 

GM3 

GM4 

AL2 

AHl 

GLl 

GHl 

10 

ALl 

GM2 

GM3 

GM4 

GMl 

AL2 

AHl 

GLl 

GHl 

11 

ALl 

GM3 

GM4 

GMl 

GM2 

AL2 

AHl 

GLl 

GHl 

ALl  and  AL2  are  similar  low  traffic  volume  ACY  scenarios 
AHl  is  a  high  traffic  volume  ACY  scenario 

GMl  -  GM4  are  similar  medium  traffic  volume  generic  sector  scenarios 
GLl  is  a  low  traffic  volume  generic  sector  scenario 
GHl  is  a  high  traffic  volume  generic  sector  scenario 


The  method  selected  to  assess  controller  workload  was  ATWIT  (Stein,  1985),  ATWIT  provides 
an  unobtrusive  and  reliable  means  for  collecting  participants’  ratings  of  workload  as  they  control 
traffic.  In  the  present  study,  a  touch  screen  was  used  to  present  the  workload  rating  scale  and 
record  the  controllers’  responses.  Controllers  were  instructed  to  indicate  their  current  workload 
by  pressing  one  of  the  touch  screen  buttons  labeled  from  1  (very  low)  to  10  (very  high).  The 
device  was  configured  to  query  the  controller  every  5  minutes.  The  controller  had  20  seconds  to 
respond  by  pressing  one  of  the  buttons.  If  they  were  too  busy  to  respond  within  20  seconds,  the 
maximum  workload  rating  of  10  was  recorded  by  default.  In  almost  every  instance,  controllers 
were  able  to  respond  within  the  allotted  time. 

4.  RESULTS. 

4.1  OVERVIEW. 

The  results  of  this  experiment  will  be  reported  in  sections  4.2  and  4.3.  Section  4.2  will  discuss 
analyses  conducted  on  the  measures  collected  on  the  first-day  training  sessions.  This  approach 
will  rely  heavily  on  a  one-way  Analysis  of  Variance  (ANOVA)  on  successive  trials  on  the  generic 
sector.  This  will  examine  the  extent  to  which  system  effectiveness  variables,  workload  ratings, 
and  expert  assessments  of  performance  changed  as  the  controllers  became  more  familiar  with  the 
generic  sector. 

Section  4.3  results  will  rely  heavily  on  correlational  relationships  between  the  two  sectors  with 
respect  to  system  effectiveness  variables,  workload  ratings,  and  expert  assessments  of 
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performance.  Dependent  measures  will  be  analyzed  to  determine  the  correlation  between 
performance  scores  on  the  generic  and  home  sectors.  In  addition,  these  correlations  will  be 
broken  down  by  high  and  low  traffic  volume.  This  is  done  because  it  has  been  well  established 
that  ATC  performance  and  task  load  change  depends  on  the  volume  of  traffic  (Davis,  Danahar,  & 
Fischl,  1963;  Buckley,  et  al.,  1976;  Buckley,  et  al.,  1983;  Stein,  1985;  Bisseret,  1971;  Coeterier, 
1971).  In  addition,  some  performance  variables  may  only  manifest  themselves  under  certain 
traffic  conditions.  For  example,  the  over-the-shoulder  rating  attribute  of  managing  multiple  tasks 
may  not  be  present  in  low  volume  scenarios  where  the  task  load  is  light.  However,  in  high 
volume  scenarios  where  task  load  is  increased,  there  may  be  more  of  an  opportunity  for  an 
evaluator  to  observe  the  controller  performing  multiple  tasks. 

Section  4.4  will  summarize  the  feedback  that  controllers  provided  about  the  experiment,  and  the 
results  of  the  final  questionnaire  will  be  presented  and  discussed.  The  final  questionnaire  provided 
another  means  for  evaluating  the  generic  sector  since  many  of  the  comments  centered  on  how 
representative  the  generic  sector  was  of  a  terminal  environment,  the  effectiveness  of  the  training 
booklet,  the  effectiveness  of  the  hands-on  training,  and  the  realism  of  the  simulation. 

4,2  PRACTICE  AND  LEARNING  EFFECTS  ASSOCIATED  WITH  THE  GENERIC 
SECTOR. 

The  results  of  a  series  of  one-way  ANOVAs  examining  system  effectiveness  variables,  ATC 
performance,  and  workload  is  reported  in  this  section.  A  one-way  ANOVA  is  a  formal  statistical 
technique  for  detecting  differences  between  multiple  levels  of  a  single  variable.  In  this  portion  of 
the  experiment,  the  independent  variable  examined  is  practice  as  measured  by  multiple  trials.  The 
multiple  trials  are  the  four  generic  sector  runs  each  controller  worked.  If  there  are  significant 
differences  between  the  earlier  and  later  trials,  with  respect  to  the  dependent  measures,  then  the 
results  would  strongly  suggest  that  learning  occurred.  However,  lack  of  a  significant  result  may 
have  multiple  interpretations,  as  the  dependent  measure  may  lack  sensitivity  to  learning  and  more 
trials  may  be  needed  before  a  learning  effect  can  be  detected  statistically.  The  one-way  ANOVAs 
were  based  on  44  observations  (1 1  participants  times  4  scenarios  per  participant). 

The  means  and  results  of  the  one  way  ANOVAs  conducted  on  the  performance  measures  are 
reported  in  table  4.  Only  a  subset  of  those  measures  collected  were  submitted  for  formal  analysis. 
Many  of  the  measures  collected  were  nearly  all  zero  values,  such  as  number  of  delays  or  number 
of  conflicts,  or  they  displayed  extremely  low  variance  between  controllers.  An  example  of  this 
type  of  situation  is  the  number  of  aircraft  handled.  Nearly  every  controller  handled  all  the  aircraft 
presented  in  the  traffic  sample.  Since  this  was  an  exploratory  study,  a  probability  value  of  p  <  .10 
was  used  as  an  indicator  for  what  might  appear  in  a  larger  sample. 

As  shown,  most  of  the  performance  measures  showed  a  high  degree  of  stability  and  did  not 
significantly  change  from  trial  to  trial,  with  the  exception  of  the  total  distance  flown  and  time 
under  control.  These  measures  were  significantly  lower  in  magnitude  by  the  fourth  trial.  Average 
ATWIT  ratings  are  also  included  in  this  table.  These  numbers  were  calculated  by  averaging  the 
12  ATWIT  ratings  for  each  run  to  form  a  summary  score  for  each  trial.  Planned  comparisons  of 
trial  one  versus  trial  four  showed  that  average  ATWIT  ratings  were  significantly  lower  in 
magnitude  by  the  fourth  trial  (p  <  .01). 
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TABLE  4.  MEANS  AND  SIGNIFICANCE  VALUES  FOR  SYSTEM  EFFECTIVENESS 
VARIABLES  AND  ATWIT  FOR  FOUR  MEDIUM  GENERIC  SECTOR  RUNS 

(N=ll) 


VARIABLE 

TRIAL  1 

TRIAL  2 

TRIAL  3 

TRIAL  4. 

SIG 

CLUSTERING 

3.90 

3.7 

4.05 

3.58 

N.S. 

ALTITUDE  CHANGES 

60 

62 

65 

61 

N.S. 

HEADING  CHANGES 

58 

54 

58 

56 

N.S. 

SPEED  CHANGES 

6 

6 

5 

4 

N.S. 

NO.  COMMUNICATIONS 

156 

151 

151 

146 

N.S. 

DUR.  COMMUNICATIONS 

703 

687 

706 

664 

N.S. 

PILOT  KEYSTROKES 

896 

874 

919 

882 

N.S. 

TIME  UNDER  CONTROL 

17109 

17032 

17519 

16762 

p  <  .10 

DISTANCE  FLOWN  (miles) 

1047 

1041 

1078 

1025 

p<.05 

ARRIVAL  INTERVAL  (sec) 

153 

152 

152 

155 

N.S. 

AVERAGE  ATWIT  RATING 

4.90 

4.40 

4.40 

3.80 

p<.01 

N.S.  -  Not  Significant 

The  means  and  results  of  ANOVAs  conducted  on  the  over-the-shoulder  ratings  are  presented  in 
table  5.  These  ratings  are  based  on  a  10-point  scale  where  a  rating  of  1-3  indicates  generally  poor 
performance,  a  rating  of  4-7  indicates  satisfactory  performance,  and  a  rating  of  8-10  indicates 
superior  performance.  These  results  also  indicate  an  average  satisfactory  performance  for  every 
performance  dimension  as  measured  by  the  rating  form  on  the  generic  sector.  These  ratings  did 
increase  slightly  over  trials  for  every  performance  dimension,  but  not  significantly. 

The  cell  means  and  results  of  ANOVAs  conducted  on  the  post-scenario  questionnaire  ratings  are 
presented  in  table  6.  These  ratings  are  on  a  10-point  scale,  with  the  exception  of  the  workload 
scale  which  was  based  on  a  12-point  scale.  Most  of  the  items  were  phrased  in  terms  of  how  well 
the  controller  performed  on  a  particular  dimension  with  a  1  indicating  not  very  well  and  a  10 
indicating  extremely  well.  Exceptions  to  the  not  very  well/extremely  well  dimension  include 
thepractice  item  which  asked  the  controllers  how  much  they  felt  they  could  improve  with  practice, 
with  a  higher  score  indicating  a  greater  amount  of  improvement  potential.  These  rating 
dimensions  were  adapted  from  the  current  FAA  over-the-shoulder  rating  form. 

4.3  CORRELATION  CO-EFFICIENTS  BETWEEN  THE  GENERIC  SECTOR  AND  ACY 
SECTOR  WITH  RESPECT  TO  PERFORMANCE  MEASURES. 

The  relationship  between  performance  on  the  generic  sector  and  the  ACY  sector  was  assessed 
through  correlational  analysis.  Scores  collected  from  the  ACY  sector  high-  and  low-volume 
traffic  runs  were  correlated  with  scores  from  the  generic  sector  high-  and  low-volume  runs, 
respectively.  A  correlational  analysis  is  a  formal  statistical  technique  for  calculating  the  degree  to 
which  two  variables  relate  or  covary.  The  results  of  the  analysis  produce  a  correlation  coefficient 
which  ranges  form  -1.0  to  +1.0  and  indicates  the  strength  and  direction  of  the  relationship 
between  two  variables.  A  coefficient  of  0.0  means  that  no  relationship  exists,  while  -1.0  and  +1 .0 
indicates  a  perfect  relationship.  A  positive  coefficient  means  that  as  the  value  of  one  variable 
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TABLE  5.  MEANS  AI^  SIGNIFICANCE  VALUES  FOR  THE  OVER-THE-SHOULDER 
RATINGS  FOR  FOUR  GENERIC  SECTOR  RUNS 

(N=ll) 


VARIABLE 

TRIAL  1 

TRIAL  2 

TRIALS 

TRIAL  4 

SIGNIFICANCE 

COMMUNICATION 

5.91 

6.18 

6.55 

6.73 

N.S. 

MANAGING  TASKS 

6.27 

6.27 

6.73 

6.82 

N.S. 

TECH.  KNOWLEDGE 

6.27 

5.82 

6.18 

6.27 

N.S. 

STRESS 

6.27 

6.55 

6.55 

6.73 

N.S. 

ATTENTION 

6.09 

6.18 

6.36 

6.55 

N.S. 

PRIORITIZATION 

5.64 

6.27 

6.64 

6.55 

N.S. 

TRAFFIC  FLOW 

6.00 

5.64 

6.45 

6.36 

N.S. 

ADAPTABILITY 

5.91 

6.27 

6.45 

6.64 

N.S. 

TABLE  6.  MEANS  AND  SIGNIFICANCE  VALUES  FOR  THE  POST-SCENARIO 
QUESTIONNAIRE  RATINGS  FOR  FOUR  GENERIC  SECTOR  RUNS  (N  =  1 1) 


VARIABLE 

TRIAL  1 

TRIAL  2 

TRIALS 

TRIAL  4 

TRIAL  1  VS  4 

WORKLOAD 

6.91 

6.64 

6.55 

5.45 

p<.01 

INFO  EXCHANGE 

6.45 

7.55 

7.18 

8.27 

p<.01 

ATTENTION 

7.09 

7.27 

7.27 

8.09 

N.S. 

PRIORITIZATION 

6.55 

7.64 

7.64 

8.27 

p<.05 

TECH.  KNOW. 

7.36 

7.91 

8.00 

8.18 

N.S. 

PHRASEOLOGY 

7.27 

7.91 

7.45 

8.36 

p<.01 

PRE-PLANNING 

6.55 

7.27 

7.45 

8.18 

p<.01 

TRAFFIC  FLOW 

7.00 

7.64 

7.18 

8.09 

p<.05 

STRESS 

5.55 

5.55 

6.09 

4.18 

N.S. 

SAFETY 

7.55 

7.55 

7.55 

8.45 

N.S. 

PRACTICE 

7.36 

6.73 

6.27 

5.27 

p<.05 

increases  the  value  of  the  second  variable  increases  as  well.  A  negative  coefficient  means  that  as 
the  value  of  one  variable  increases  the  value  of  the  other  variable  decreases.  Strong  significant 
correlation  co-efficients  suggest  that  performance  on  the  generic  sector  is  related  to  performance 
on  the  "home"  sector.  Specifically,  a  high  positive  correlation  indicates  that  if  a  controller 
performed  well  on  a  performance  dimension  on  the  generic  sector,  he  or  she  also  performed  well 
on  this  dimension  for  the  ACY  sector.  This  same  correlation  would  also  indicate  that  if  a 
controller  did  not  perform  well  on  a  performance  dimension  on  the  generic  sector,  he  or  she  also 
did  not  perform  well  on  this  dimension  for  the  ACY  sector. 

Table  7  shows  the  results  for  10  of  the  system  effectiveness  variables  measured  in  the  simulation. 
This  table  also  includes  average  ATWIT  ratings.  The  results  show  significant  correlations  for  the 
number  of  altitude  changes  and  for  the  duration  of  communications  for  the  high-volume  runs  and 
the  number  of  altitude  changes  and  the  number  of  simulation  pilot  keystrokes  for  the  low-volume 
runs.  In  addition,  when  high  and  low  traffic  volume  runs  are  averaged,  the  average  scores  are 
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TABLE  7.  CORRELATION  CO-EFFICIENTS  AND  SIGNIFICANCE  VALUES  FOR 
SYSTEM  EFFECTIVENESS  VARIABLES  FOR  ACY  AND  GENERIC  SECTORS 


VARIABLE 

HIGH  VOLUME 
(N  =  10) 

LOW  VOLUME 
(N-9) 

AVERAGED 
(N  =  9) 

CLUSTERING 

-0.12 

0.39 

0.52 

ALTITUDE  CHANGES 

0.63** 

0.80* 

0.84* 

HEADING  CHANGES 

0.11 

-0.19 

0.07 

SPEED  CHANGES 

0.46 

0.47 

0.63*** 

NO.  COMMUNICATIONS 

0.50 

0.57 

0.74** 

DUR.  COMMUNICATIONS 

0.58*** 

0.27 

0.74** 

NO.  PILOT  KEYSTROKES 

0.20 

0.76** 

0.49 

TIME  UNDER  CONTROL  (sec) 

0.08 

0.61*** 

0.33 

DISTANCE  FLOWN  (miles) 

0.09 

0.58*** 

0.37 

AVERAGE  ATWIT 

0.80* 

0.90* 

0.90* 

*p<.01 

**p<.05 

*»»p<,10 


correlated  for  each  system  effectiveness  variable.  These  correlation  co-efficients  are  much  higher 
and  more  consistent  across  the  variables  then  for  the  high  or  low  volume  data,  respectively. 
Correlation  co-efficients  which  could  be  significantly  different  from  zero  with  a  p  <  .  10  were 
included,  since  this  was  an  exploratoiy  study. 

Table  8  shows  the  results  from  the  correlational  analyses  from  the  over-the-shoulder  rating  form 
(appendix  C).  For  the  high-volume  runs,  five  of  the  eight  rating  categories  are  significantly 
correlated,  indicating  a  positive  relationship  between  performance  on  the  ACY  sector  and  the 
generic  sector.  The  significant  correlation  co-efficients  range  from  r=.64  to  r=.80,  indicating  a 
moderate  to  high  relationship  between  performance  on  these  sectors.  For  the  low  volume  runs, 
only  the  communication  category  was  significantly  related. 


TABLE  8.  CORRELATION  CO-EFFICIENTS  AND  SIGNIFICANCE  VALUES  FOR  OVER- 
THE-SHOULDER  RATINGS  FOR  ACY  AND  GENERIC  SECTORS 


VARIABLE 

HIGH  VOLUME 
(N=10) 

LOW  VOLUME 
(N=  10) 

COMMUNICATION 

0.71** 

0.68** 

MANAGING  TASKS 

0.80* 

0.00 

TECHNICAL  KNOWLEDGE 

0.59 

0.25 

STRESS 

0.64** 

0.40 

ATTENTION  &  VIGILANCE 

0.64** 

0.16 

PRIORITIZATION 

0.39 

-0.38 

TRAFFIC  FLOW 

0.38 

-0.19 

ADAPTABILITY 

0.71** 

0.71 

•p  <  .01 

**  p  <  .05 
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Table  9  shows  the  results  from  the  correlational  analyses  from  the  post-scenario  questionnaire 
ratings  that  the  controllers  completed  at  the  end  of  each  scenario  run.  These  rating  categories 
were  derived  from  the  current  FAA  over-the-shoulder  rating  form.  The  results  indicate  high 
positive  correlation  co-efficients  for  every  rating  category  with  the  exception  of  prioritization  for 
the  high-volume  runs.  Five  of  the  categories  were  significantly  correlated  for  the  low-volume 
runs. 


TABLE  9.  CORRELATION  CO-EFFICIENTS  AND  SIGNIFICANCE  VALUES  FOR  THE 
POST-SCENARIO  QUESTIONNAIRE  RATINGS  FOR  ACY  AND  GENERIC  SECTORS 


VARIABLE 

HIGH  VOLUME 
(N=  10) 

,  LOW  VOLUME 
(N=10) 

WORKLOAD 

0.77* 

0.72** 

INFO  EXCHANGE 

0.91* 

0.50 

ATTENTION 

0.66* 

0.30 

PRIORITIZATION 

0.44 

0.38 

TECH.  KNOWLEDGE 

0,75* 

0.54 

PHRASEOLOGY 

0.85* 

0.72* 

PRE-PLANNING 

0.86* 

0.85* 

TRAFFIC  FLOW 

0.91* 

0.80* 

STRESS 

0.71** 

0.51 

SAFETY 

0.72** 

0.74** 

PRACTICE 

0.66** 

0.54 

*  P<. 

**  p  <  .05 


4,4  FINAL  QUESTIONNAIRE  COMMENTS  ON  THE  ENTIRE  EXPERIMENT. 

A  final  questionnaire  was  administered  to  each  controller  at  the  end  of  their  second  day  session. 
The  questions  requested  information  concerning  the  realism  of  the  simulation  and  the 
effectiveness  of  the  training  aids  developed  for  this  experiment.  These  comments  are  summarized 
in  table  10.  As  far  as  the  realism  of  the  simulation,  the  majority  (8  of  10  responses)  thought  the 
simulation  was  very  realistic.  The  remaining  two  controllers  thought  that,  for  the  most  part,  the 
simulation  was  realistic.  The  majority  of  the  controllers  thought  the  generic  sector  was 
representative  of  a  typical  terminal  environment  (6  of  10  responses).  The  training  booklet 
received  a  somewhat  mediocre  review.  A  number  of  controllers  thought  that  there  was  more 
information  than  needed,  although  it  was  helpful  for  learning  the  frequencies  and  general  airspace 
layout.  All  controllers  responded  positively  to  the  hands-on  training  session  they  received  during 
the  first  day  session.  The  questions  and  a  complete  summary  of  the  responses  are  listed  in 
appendix  E. 
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TABLE  10.  FINAL  QUESTIONNAIRE  COMMENTS  ON  THE  GENERIC  SECTOR 

SIMULATION 


Realism  of  the  simulation 

Number  of  Controllers  Commenting 

Very  realistic 

7 

For  the  most  part  realistic 

2 

About  90%  realistic 

1 

Representative  of  a  typical  sector 

Number  of  Controllers  Commenting 

Very  representative 

5 

O.k.  somewhat  typical 

3 

Not  too  typical 

1 

No  comment 

1 

Helpfulness  of  the  training  booklet 

Number  of  Controllers  Commenting 

Very  helpful 

1 

Somewhat  helpful 

4 

Not  very  helpful 

3 

Didn't  look  at  it 

2 

Adequacy  of  the  hands-on-training 

Number  of  Controllers  Commenting 

Yes,  it  was  adequate 

10 

No,  it  wasn't  adequate 

0 

Responsiveness  of  the  pseudo-pilots 

Number  of  Controllers  Commenting 

Excellent  job 

3 

Very  good  job 

4 

Good job 

2 

Fair  job 

1 

Intrusiveness  of  the  ATWIT  device 

Number  of  Controllers  Commenting 

Yes,  it  did  interfere  with  operations 

0 

No,  it  did  not  interfere  with  operations 

10 

5.  SUMMARY  AND  CONCLUSIONS. 

5. 1  DISCUSSION  OF  LEARNING  RATE  FOR  THE  GENERIC  SECTOR. 

Learning  rate  for  the  generic  sector  can  be  inferred  from  differences  in  the  performance  scores 
over  trials  on  the  first  day  for  the  four  performance  measurement  categories.  Air  Traffic 
Workload  Input  Technique  (ATWIT)  data  provided  the  strongest  support  for  learning  with 
significantly  lower  ATWIT  scores  by  the  last  trial.  One  explanation  for  this  finding  is  that  many 
features  of  the  sector  became  more  familiar  as  controllers  went  though  the  multiple  runs. 
Specifically,  controllers  learned  the  fix  locations,  the  routes,  the  typical  flight  plans,  and  landing 
altitudes.  As  this  information  was  learned,  it  became  more  automatic,  and  therefore,  the 
controller  did  not  have  to  expend  as  much  energy  thinking  about  these  sector  features  as  they  did 
during  the  initial  runs. 
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Two  system  effectiveness  variables,  total  time  under  control  and  total  distance  flown,  were 
significantly  lower  by  the  last  generic  run.  One  possible  explanation  for  these  results  is  that 
arrival  aircraft  were  being  handled  more  efficiently  on  the  last  run.  This  efficiency  was 
accomplished  by  closer  turn-ons  to  final  and  better  control  techniques  in  spacing  arrivals  on  the 
localizer. 

Post-scenario  questionnaire  ratings  for  information  exchange,  prioritization,  and  pre-planning 
were  also  significantly  higher  by  the  last  generic  run.  Questionnaire  ratings  of  workload  and 
stress  were  significantly  lower  by  the  last  generic  run.  These  results  also  support  the  idea  that 
controllers  became  more  familiar  with  the  features  of  the  generic  sector  in  the  later  mns.  By  the 
last  run,  when  controllers  had  learned  the  fixes,  flight  plans,  and  routes,  they  were  better  able  to 
perform  strategic  functions  such  as  prioritization  and  pre-planning.  Lower  workload  and  stress 
were  a  result  of  familiarity,  better  planning,  and  prioritization  in  the  later  runs. 

The  fact  that  performance  indicators  did  not  change  appreciably  over  the  four  runs  is  considered 
a  positive  finding  since  Genera  was  designed  for  ease  of  implementation.  Controllers  were  able 
to  learn  it  very  quickly  ar.d  work  traffic  with  no  major  complications.  Ideally,  a  generic  sector 
should  not  pose  hurdles,  but  rather  it  should  facilitate  performance.  This  sector  did  just  that. 

5.2  DISCUSSION  OF  CORRELATIONS  BETWEEN  PERFORMANCE  SCORES  ON  THE 
HOME  SECTOR  AND  THE  GENERIC  SECTOR. 

Three  of  the  four  performance  categories  showed  high  and  consistent  correlations  between  the 
home  and  Atlantic  City  International  Airport  (ACY)  sectors.  These  categories  were  ATWIT 
ratings,  over-the-shoulder  ratings,  and  post-scenario  questionnaire  ratings.  These  relationships 
suggest  that  controller  timing,  communication,  and  task  management  were  basically  the  same 
regardless  of  the  sector  configuration.  Workload,  as  measured  by  ATWIT,  was  also  highly 
correlated  between  the  two  sectors  for  both  high  and  low  traffic  runs.  This  result  suggests  that 
once  the  sector  was  learned,  the  task  load  was  basically  the  same  regardless  of  the  sector 
configuration.  The  results  indicated  that  controller  performance,  as  measured  by  ratings  and  self- 
report  measures  of  workload,  was  very  similar  in  both  sector  configurations. 

The  automated  System  Effectiveness  Measure  (SEM)  variables  showed  low  to  moderate 
correlations  between  the  home  and  generic  sectors.  This  could  mean  that  there  are  low 
relationships  between  SEMs  for  the  two  sectors,  given  the  small  sample  sizes  and  limited  number 
of  runs.  Given  the  fact  that  the  majority  of  other  data  does  correlate,  a  more  likely  hypothesis  is 
that  more  data  is  needed  before  the  SEMs  will  provide  significant  correlations.  Support  for  this 
idea  is  provided  by  the  correlational  analyses  which  used  data  which  was  averaged  over  the  high 
and  low  traffic  trials.  These  correlation  co-efficients  are  larger  in  magnitude  than  correlation  co¬ 
efficients  using  the  segregated  high  and  low  traffic  volume  data. 

There  is  another  more  logical  explanation  as  to  why  the  correlations  were  not  higher  than  they 
were.  The  researchers  on  this  project  may  have  tried  too  hard  to  make  the  generic  sector  all  it 
could  be.  They  were  concerned  with  its  generalizability  to  other  contexts  and  felt  that  it  would 
be  enough  if  it  was  similar  to  the  home  sector  and  could  evoke  relatively  comparable 
performance  patterns.  The  traffic  scenarios  were  developed  using  the  Official  Airline  Guide  as 
compared  to  those  for  the  home  sector  that  were  based  on  the  actual  traffic  seen  at  ACY.  In  an 
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effort  to  create  the  ultimate  terminal  generic  sector,  they  may  have  inadvertently  created 
differences  between  the  home  sector  and  generic  runs  that  lowered  the  performance  relationships. 
This  was  a  classic  design  artifact,  and  it  was  fortunate  that  the  correlations  were  as  high  as  they 
were.  In  a  follow  up  study,  subject  matter  experts  and  supervisory  controllers  rated  20  hours  of 
video-taped  performance  from  this  study.  They  used  a  new  evaluation  forrn  developed  for 
reliability  and  validity.  The  experts  were  able  to  transcend  the  artificial  air  traffic  scenario 
differences,  and  they  identified  much  higher  relationships  across  home  and  generic  sectors.  This 
follow-on  study  will  be  reported  at  a  later  date. 

In  general,  these  findings  support  the  idea  that  a  generic  sector,  once  learned,  can  be  used  as  a 
performance  measurement  tool  even  though  it  may  have  different  physical  features  from  the 
controller's  home  sector. 
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APPENDIX  A 

LETTERS  OF  AGREEMENT  BETWEEN  GENERIC  SECTOR  AND  ADJACENT  AIRPORTS 

Effective;  1  November  1994 

SUBJECT;  Tower  En  Route  Control  (TEC) 

TO;  Genera  International  Airport 
Texas  Regional  Airport 

1.  PURPOSE.  This  agreement  defines  Genera  International  Airport  and  Texas  Regional  Airport 
procedures  for  Instrument  Flight  Rule  (IFR)  TEC. 

2.  SCOPE.  The  TEC  procedures  contained  herein  shall  apply  unless  prior  coordination's 
affected.  These  procedures  are  effective  only  with  operational  radar.  TEC  will  be  on  an  approval 
request  basis  if  either  facility's  radar  is  out  of  service. 

3.  PROCEDURES. 

a.  TEC  is  limited  to  flights  landing  and  taking  off  within  both  approach  control's  airspace. 

b.  Radar  handoffs  are  mandatory  prior  to  entering  the  receiving  facility's  area,  either  by 
automated  procedures  or  manually  via  interphone.  Transfer  of  control  releases  control  for 
turns  of  up  to  30  degrees  fi-om  assigned  heading. 

c.  Transfer  Control  Points  (TCPs); 

1)  The  TCPs,  with  associated  routes,  altitudes,  and  frequencies  for  TEC,  are  listed 
below  and  depicted  on  attachment  1 

d.  Flow  Control. 

1)  Initiate,  update  ,and  relay  essential  information  between  facilities  to  establish  and 
ensure  an  acceptable  TEC  traffic  flow. 

2)  Wlien  flow  control  restrictions  have  been  imposed  within  the  TEC  structures, 
concerned  facilities  shall  inform  adjacent  locations. 

3)  Verbal  approval  requests  shall  be  required  prior  to  releasing  aircraft  on  the  ground 
when  delays  are  in  effect. 

4.  COORDINATION.  All  advance  flight  coordination  shall  be  via  automated  flight  progress 
strips.  In  the  event  of  NAS  computer  failure,  the  aircraft  identification,  type  aircraft,  discrete 
transponder  code,  route  destination  airport,  and  altitude  shall  have  been  relayed  to  the  receiving 
facility  prior  to  initiating  a  handoff. 
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TEXAS  TO  GENERA  (TERMINAL  EN  ROUTE  CONTROLS 


TCP 

ROUTE 

ALTITUDES 

FREQUENCY 

TGNXA 

TEX  GENNY  GEN 

3000/5000 

127.5 

TGNXA 

TEX  MIDDY  MID 

3000/5000 

127.5 

TGNXA 

TEXUPPTYUPT 

3000/5000 

127.5 

TGNXA 

TEX  DOWNY  DWN 

3000/5000 

127.5 

GENERA  TO  TEXAS  TTERMINAL  EN  ROUTE  CONTROLS 


TCP  ROUTE 


ALTITUDES  FREQUENCY 


WESTD  GEN  WESTD  TEX 

WESTD  UPT  WESTD  TEX 

WESTD  MID  WESTD  TEX 

WESTD  DWN  WESTD  TEX 


4000/6000 

128.0 

4000/6000 

128.0 

4000/6000 

128.0 

4000/6000 

128.0 

5.  ATTACHMENT 

Attachment  1. — Transfer  of  Control  Points,  Arrival/Departure  Routes  and  Frequencies. 


EDWARD  BUCKLEY 

Manager,  Genera  International  Airport 


EARL  STEIN 

Manager,  Texas  Regional  Airport 


TEX/GEN  TWR  LO  A 


EFFECTIVE;  NOVEMBER  1,  1994 


LETTER  OF  AGREEMENT 


Effective;  1  November  1994 

SUBJECT:  Tower  En  Route  Control  (TEC) 

TO:  Genera  International  Airport 
Georgia  Regional  Airport 

1 .  PURPOSE.  This  agreement  defines  Genera  International  Airport  and  Georgia  Regional 
Airport  procedures  for  Instrument  Flight  Rule  (EFR)  TEC. 

2.  SCOPE.  The  TEC  procedures  contained  herein  shall  apply  unless  prior  coordination  is 
effected.  These  procedures  are  effective  only  with  operational  radar.  TEC  will  be  on  an  approval ' 
request  basis  if  either  facility's  radar  is  out  of  service. 

3.  PROCEDURES. 


a.  TEC  is  limited  to  flights  landing  and  taking  off  within  both  approach  control's  airspace. 

b.  Radar  handoffs  are  mandatory  prior  to  entering  the  receiving  facility's  area,  either  by 
automated  procedures  or  manually,  via  interphone.  Transfer  of  control  releases  control  for 
turns  of  up  to  thirty  (30)  degrees  from  assigned  heading. 


c.  Transfer  Control  Points  (TCPs); 

1)  The  TCPs,  with  associated  routes,  altitudes  and  frequencies  for  TEC  are  listed  below 
and  depicted  on  attachment  1 . 

GEORGIA  TO  GENERA  fTERMINAL  EN  ROUTE  CONTROL! 


TCP 

ROUTE 

ALTITUDES 

FREQUENCY 

GGNXA 

GEO  GENNY  GEN 

3000/5000 

128.5 

GGNXA 

GEO  DOWNY  DWN 

3000/5000 

128.5 

GGNXA 

GEO  MTDDYMID 

3000/5000 

128.5 

GGNXA 

GEOGENUPPTYUPT 

3000/5000 

128.5 

GENERA  TO  GEORGIA  f TERMINAL  EN  ROUTE  CONTROL! 


TCP 

ROUTE 

ALTITUDES 

FREQUENCY 

SWEDD 

GEN  SWEDD  GEO 

4000/6000 

129.0 

SWEDD 

DWN  SWEDD  GEO 

4000/6000 

129.0 

SWEDD 

MID  SWEDD  GEO 

.  4000/6000 

129.0 

SWEDD 

UPT  SWEDD  GEO 

4000/6000 

129.0 
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5.  ATTACHMENT 


Attachment  1. — Transfer  of  Control  Points,  Arrival/Departure  Routes,  and  Frequencies. 


EDWARD  BUCKLEY 

Manager,  Genera  International  Airport 


PAUL  STRINGER 

Manager,  Georgia  Regional  Airport 


GEO/GEN  TWR  LO A  EFFECTIVE:  NOVEMBER  1,1994 
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LETTER  OF  AGREEMENT 


Effective:  1  November  1994 

SUBJECT;  Tower  En  Route  Control  (TEC) 

TO;  Genera  International  Airport 
Maine  Regional  Airport 

1.  PURPOSE.  This  agreement  defines  Genera  International  Airport  and  Maine  Regional  Airport 
procedures  for  Instrument  Flight  Rule  (IFR)  TEC. 

2.  SCOPE.  The  TEC  procedures  contained  herein  shall  apply  unless  prior  coordination  is 
effected.  These  procedures  are  effective  only  with  operational  radar.  TEC  will  be  on  an  approval 
request  basis  if  either  facility's  radar  is  out  of  service. 

3.  PROCEDURES. 

a.  TEC  is  limited  to  flights  landing  and  taking  off  within  both  approach  controls’  airspace. 

b.  Radar  handoffs  are  mandatory  prior  to  entering  the  receiving  facility's  area,  either  by 
automated  procedures  or  manually,  via  interphone.  Transfer  of  control  releases  control  for 
turns  of  up  to  thirty  (30)  degrees  from  assigned  heading. 

c.  Transfer  Control  Points  (TCPs): 

1)  The  TCPs,  with  associated  routes,  altitudes,  and  fi-equencies  for  TEC  are  listed 
below  and  depicted  on  ATTACHMENT  1. 


MAINE  TO  GENERA  TTERMINAL  EN  ROUTE  CONTROLS 


TCP 

ROUTE 

ALTITUDES 

FREQUENCIES 

MGNXA 

MAI  GENNY  GEN 

3000/5000 

129.5 

MGNXA 

MAI  GEN  DOWNY  DWN 

3000/5000 

129.5 

MGNXA 

MAI  MIDDY  MID 

3000/5000 

129.5 

MGNXA 

MAI  UPPTY  UPT 

3000/5000 

129.5 

GENERA 

TO  MAINE  fTERMINAL  EN  ROUTE  CONTROL! 

TCP 

ROUTE 

ALTITUDES 

FREQUENCIES 

NEAST 

GEN  NEAST  MAI 

4000/6000 

130.0 

NEAST 

DWN  GEN  NEAST  MAI 

4000/6000 

130.0 

NEAST 

MID  NEAST  MAI 

4000/6000 

130.0 

NEAST 

UPT  NEAST  MAI 

4000/6000 

130.0 
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d.  Flow  Control. 


1)  Initiate,  update,  and  relay  essential  information  between  facilities  to  establish  and 
ensure  an  acceptable  TEC  traffic  flow. 

2)  When  flow  control  restrictions  have  been  imposed  within  the  TEC  structures, 
concerned  facilities  shall  inform  adjacent  locations. 

3)  Verbal  approval  requests  shall  be  required  prior  to  releasing  aircraft  on  the  ground 
when  delays  are  in  effect. 

4.  COORDINATION.  All  advanced  flight  coordination  shall  be  via  automated  flight  progress 
strips.  In  the  event  of  National  Airspace  System  (NAS)  computer  failure,  the  aircraft 
identification,  type  of  aircraft,  discrete  transponder  code,  route  destination  airport,  and  altitude 
shall  have  been  relayed  to  the  receiving  facility  prior  to  initiating  a  handoff 

5.  ATTACHMENT 

Attachment  I. — Transfer  of  Control  Points,  Arrival/Departure  Routes,  and  Frequencies. 


EDWARD  BUCKLEY 

Manager,  Genera  International  Airport 


JERRY  GUTTMAN 
Manager,  Maine  Regional  Airport 


MAI/GEN  TWRLOA 


EFFECTIVE;  NOVEMBER  I,  1994 
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LETTER  OF  AGREEMENT 


Effective:  1  November  1994 

SUBJECT;  Tower  En  Route  Control  (TEC) 

TO;  Genera  International  Airport 
Atlantic  Regional  Airport 

1 .  PURPOSE.  This  agreement  defines  Genera  International  Airport  and  Atlantic  Regional 
Airport  procedures  for  Instrument  Flight  Rule  (IFR)  TEC. 

2.  SCOPE.  The  TEC  procedures  contained  herein  shall  apply  unless  prior  coordination  is 
effected.  These  procedures  are  effective  only  with  operational  radar.  TEC  will  be  on  an  approval 
request  basis  if  either  facility's  radar  is  out  of  service. 

3.  PROCEDURES. 


a.  TEC  is  limited  to  flights  landing  and  taking  off  within  both  approach  control's  airspace. 

b.  Radar  handoffs  are  mandatory  prior  to  entering  the  receiving  facility's  area,  either  by 
automated  procedures  or  manually,  via  interphone.  Transfer  of  control  releases  control  for 
turns  of  up  to  thirty  (30)  degrees  from  assigned  heading. 

c.  Transfer  Control  Points  (TCPs): 

1)  The  TCPs,  with  associated  routes,  altitudes,  and  frequencies  for  TEC  are  listed 
below  and  depicted  on  ATTACHMENT  1. 


ATLANTIC  TO  GENERA  CTERMINAL  EN  ROUTE  CONTROL! 


TCP 

ROUTE 

ALTITUDES 

FREQUENCY 

AGNXA 

ALT  GENNY  GEN 

3000/5000 

130.5 

AGNXA 

ALT  DOWNY  DWN 

3000/5000 

130.5 

AGNXA 

ALT  GEN  MIDDY  MID 

3000/5000 

130.5 

AGNXA 

ALTUPPTYUPT 

3000/5000 

130.5 

GENERA  TO  ATLANTIC  (TERMINAL  EN  ROUTE  CONTROL! 


TCP 

ROUTE 

ALTITUDES 

FREQUENCY 

EASTD 

GEN  EASTD  ALT 

4000/6000 

131.0 

EASTD 

DWN  EASTD  ALT 

4000/6000 

131.0 

EASTD 

MED  GEN  EASTD  ALT 

4000/6000 

131.0 

EASTD 

UPT  EASTD  ALT 

4000/6000 

131.0 
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d.  Flow  Control. 

1)  Initiate,  update,  and  relay  essential  information  between  facilities  to  establish  and 
ensure  an  acceptable  TEC  traffic  flow. 

2)  When  flow  control  restrictions  have  been  imposed  within  the  TEC  structures, 
concerned  facilities  shall  inform  adjacent  locations. 

3)  Verbal  approval  requests  shall  be  required  prior  to  releasing  aircraft  on  the  ground 
when  delays  are  in  effect. 

4.  COORDINATION.  All  advanced  flight  coordination  shall  be  via  automated  flight  progress 
strips.  In  the  event  of  NAS  computer  failure,  the  aircraft  identification,  type  aircraft,  discrete 
transponder  code,  route  destination  airport,  and  altitude  shall  have  been  relayed  to  the  receiving 
facility  prior  to  initiating  a  handoff. 

5.  ATTACHMENT 

Attachment  1. — Transfer  of  Control  Points,  Arrival/Departure  Routes,  and  Frequencies. 


EDWARD  BUCKLEY 

Manager,  Genera  International  Airport 


DENNIS  FILLER 

Manager,  Atlantic  Regional  Airport 


ALT/GEN  TWRLO  A 


EFFECTIVE;  NOVEMBER  1,  1994 
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APPENDIX  B 


LETTERS  OF  AGREEMENT  BETWEEN  GENERIC  SECTOR  AND  EN  ROUTE  CENTERS 

NORTH  CENTRAL  CENTER  AND  GENERA  INTERNATIONAL  AIRPORT 

LETTER  OF  AGREEMENT 


Effective:  November  1,  1994 

SUBJECT:  APPROACH  CONTROL  SERVICE 

1 .  PURPOSE:  To  establish  Genera  Approach  Control  boundaries  and  define  the  coordination 
necessary  to  exercise  approach  control  service. 

2.  DELEGATION  OF  AUTHORITY:  North  Central  Center  (ZNC)  delegates  to  Genera  Tower 
(GEN)  authority  and  responsibility  for  control  of  Instrument  Flight  Rule  (IFR)  arrival,  departure, 
and  tower  en  route  aircraft  in  controlled  airspace  within  the  Approach  Control  Area,  as  described 
herein. 

3.  SCOPE:  The  procedures  contained  herein  shall  apply  unless  prior  coordination  has  been 
effected. 

4.  GENERAL:  Minimum  separation  between  aircraft  when  transfer  of  control  is  accomplished 
shall  be  at  least  5  miles  constant  or  increasing. 

5.  RADAR  HANDOFFS:  Radar  handoffs  are  mandatory  prior  to  the  aircraft  entering  the 
receiving  facilities’  airspace. 

6.  COORDINATION :  Verbal  coordination  is  not  required  for  any  aircraft  having  an  Automated 
Terminal  Radar  System  (ARTS)-  or  Flight  Data  Entry  Printout  (FDEP)-  generated  flight  plan.  In 
the  event  of  ZNC  computer  or  GEN  FDEP  failure,  the  receiving  facility  shall  be  informed  of  the 
incoming  or  outgoing  flights  flight  plan  well  in  advance  of  the  flights  arrival  or  departure. 

7.  PROCEDURES: 

a.  DEPARTURE  CONTROL 

1)  GEN  Approach  Control  shall  initiate  departure  clearances  to  all  aircraft  without 
calling  the  center. 

2)  GEN  shall  clear  departure  aircraft  to  altitudes  within  the  Approach  Control 
Airspace  (17000  and  below). 

3)  Departure  aircraft  shall  be  established  on  assigned  routes  by  GEN  prior  to 
penetrating  ZNC  airspace,  as  shown  on  Attachment  1. 

4)  Aircraft  requesting  altitude/flight  level  above  the  initial  clearance  altitude  (17000) 
shall  be  told  to  expect  further  clearance  to  requested  altitude/flight  level  ten  (10) 
minutes  after  departure. 
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b.  ARRIVAL  CONTROL 


1)  All  arrivals  shall  be  assigned  airport  clearances  via  routings  depicted  in  Attachment 
2. 

EXCEPTIONS: 

a)  When  GEN  radar  is  inoperative. 

b)  When  arrival  delays  are  anticipated. 

2)  All  arrivals  shall  be  cleared  to  descend  to  8000  providing  radar  handofF  and 
communications  transfer  is  made  prior  to  the  arrival  aircraft  entering  GEN  airspace. 

3)  The  tower  shall  have  control  for  descent  and  30  degree  turns  right  or  left  of  course 
after  transfer  of  control  has  been  accomplished. 

4)  All  arrival  aircraft  shall  have  a  speed  restriction  of  250  kt  when  the  hourly  arrival 
rate  exceeds  25  aircraft. 

8.  ATTACHMENTS: 

a.  Attachment  1 —  Departure  routes,  Frequencies,  and  Transfer  of  Control  Points. 

b.  Attachment  2 —  Arrival  routes,  Frequencies,  and  Transfer  of  Control  Points. 


ED  BUCKLEY 
Air  Traffic  Manager 
Genera  International  Tower 


KATHY  MANN 
Air  Traffic  Manager 
North  Central  Center 


ZNC/GENTWRLOA 


EFFECTIVE  NOVEMBER  1,  1994 
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SOUTH  CENTRAL  CENTER  AND  GENERA  INTERNATIONAL  AIRPORT 

LETTER  OF  AGREEMENT 


Effective;  November  1,  1994 

SUBJECT:  APPROACH  CONTROL  SERVICE 

1 .  PURPOSE:  To  establish  Genera  Approach  Control  boundaries  and  define  the  coordination 
necessary  to  exercise  approach  control  service. 

2.  DELEGATION  OF  AUTHORITY:  South  Central  Center  (ZSC)  delegates  to  Genera  Tower 
(GEN),  authority  and  responsibility  for  control  of  IFR  arrival,  departure  and  tower  en  route 
aircraft  in  controlled  airspace  within  the  Approach  Control  Area  as  described  herein. 

3.  SCOPE:  The  procedures  contained  herein  shall  apply  unless  prior  coordination  has  been 
effected. 

4.  GENERAL:  Minimum  separation  between  aircraft  when  transfer  of  control  is  accomplished 
shall  be  at  least  five  (5)  miles  constant  or  increasing. 

5.  RADAR  HANDOFFS:  Radar  handoflfs  are  mandatory  prior  to  the  aircraft  entering  the 
receiving  facilities’  airspace. 

6.  COORDINATION :  Verbal  coordination  is  not  required  for  any  aircraft  having  an  ARTS  or 
FDEP  generated  flight  plan.  In  the  event  of  ZSC  computer  or  GEN  FDEP  failure,  the  receiving 
facility  shall  be  informed  of  the  incoming  or  outgoing  flights  flight  plan  well  in  advance  of  the 
flights  arrival  or  departure. 

7.  PROCEDURES: 

a.  DEPARTURE  CONTROL 


1)  GEN  Approach  Control  shall  initiate  departure  clearances  to  all  aircraft  without 
calling  the  center. 

2)  GEN  shall  clear  departure  aircraft  to  altitudes  within  the  Approach  Control 
Airspace  (17000  and  below). 

3)  Departure  aircraft  shall  be  established  on  assigned  routes  by  GEN  prior  to 
penetrating  ZSC  airspace  as  shown  on  Attachment  1 . 

4)  Aircraft  requesting  altitude/flight  level  above  the  initial  clearance  altitude  (17000) 
shall  be  told  to  expect  further  clearance  to  requested  altitude/  flight  level  ten  (10) 
minutes  after  departure. 
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b.  ARRIVAL  CONTROL 


1)  All  arrivals  shall  be  assigned  airport  clearances  via  routings  depicted  in  attachment 

2. 

EXCEPTIONS: 

a)  When  GEN  radar  is  inoperative. 

b)  When  arrival  delays  are  anticipated. 

2)  All  arrivals  shall  be  cleared  to  descend  to  (8000)  providing  radar  handofF  and 
communications  transfer  is  made  prior  to  the  arrival  aircraft  entering  GEN  airspace. 

3)  The  tower  shall  have  control  for  descent  and  30  degree  turns  right  or  left  of  course 
after  transfer  of  control  has  been  accomplished. 

4)  All  arrival  aircraft  shall  have  a  speed  restriction  of  250  kt  when  the  hourly  arrival 
rate  exceeds  25  aircraft. 

8.  ATTACHMENTS: 


a.  Attachment  1  —  Departure  Routes,  Frequencies  and  Transfer  of  Control  Points. 

b.  Attachment  2  —  Arrival  routes,  Frequencies  and  Transfer  of  Control  Points. 


ED  BUCKLEY 
Air  Traffic  Manager 
Genera  International  Tower 


RANDY  SOLLENBERGER 
Air  Traffic  Manager 
South  Central  Center 


ZSC/GENTWRLOA 


EFFECTIVE:  NOVEMBER  1,  1994 
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APPENDIX  C 


SACHA  OVER-THE-SHOULDER  RATING  FORM 


ID# 

SCENARIO  # 


COMMUNICATING  AND  INFORMING 


USES  CLEAR  CONCISE  ACCURATE  LANGUAGE  TO  GET  MESSAGE  ACROSS  UNAMBIGUUOUSLY,  TALKING  ONLY 
WHEN  NECESSARY  AND  APPROPRIATE;  EMPLOYING  PROPER  PHRASEOLOGY  TO  ENSURE  ACCURATE 
COMMUNICATION  ;  NOTIFYING  PILOTS/CO^fTROLLER S/OTHER  PERSONNEL  OF  INFORMATION  THAT  MIGHT 
AFFECT  THEM  AS  APPROPRIATE;  ISSUING  ADVISORIES  AND  ALERTS  TO  APPROPRIATE  PARTIES;  LISTENING 
CAREFULLY  TO  REQUESTS  AND  INSTRUCTIONS  AND  ENSURING  THAT  THEY  ARE  UNDERSTOOD;  ATTENDING  TO 
READBACKS  AND  ENSURING  THAT  THEY  ARE  ACCURATE. 


Is  consistently  too  wordy, 
imprecise  in  phraseology,  or 
uses  slang  Inappropriately 
during  transmissions  to 
pilots  and  other  controllers 


Is  careless  about  informing 
pilots  concerning  circum¬ 
stances  that  affect  them  such 
as  weather,  nearby  traffic  etc. 


Often  fails  to  ensure  that  own 
instructions  are  understood;  is 
not  very  good  at  picking  up  on 
errors  in  pilot  readbacks  of 
clearances,  course  changes, 
etc. 


Radio  and  interphone  com¬ 
munications  are  usually  easy 
to  understand;  at  times,  may 
be  somewhat  wordy  or  use 
ambiguous  phraseology  on 
the  air 

Is  normally  good  at  informing 
pilots  about  situations  and 
conditions  that  affect  them 
(e.g.  safety  related  items) 


For  the  most  part  checks  to 
be  certain  that  own  Instructions 
are  understood;  only  occasion¬ 
ally  fails  to  pick  up  on  inac¬ 
curate  readbacks  from  pilots 


Always  uses  clear,  consise 
phraseology  when  talking  to 
pilots  or  other  controllers;  Is 
very  easy  to  understand 


Consistently  provides  pilots 
with  the  information  they  neec^ 
such  as  timely  safety  alerts, 
weather  advisories,  warnings 
about  unpublished 
obstructions 

Always  ensures  that  own  In¬ 
structions  are  clearly  under¬ 
stood;  pays  careful  attention 
to  pilot  readbacks  of 
clearances 
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MANAGING  MULTIPLE  TASKS 


KEEPING  TRACK  OF  A  LARGE  NUMBER  OF  AIRCRAFT/EVENTS  AT  ONE  TIME;  CONDUCTING  TWO  OR  MORE  TASKS 
SIMULTANEOUSLY  ;  REMEMBERING  AND  KEEPING  TRACK  OF  AIRCRAFT  AND  THEIR  POSITIONS;  REMEMBERING 
WHAT  YOU  WERE  DOING  AFTER  AN  INTERRUPTION;  RETURNING  TO  WHAT  YOU  WERE  DOING  AFTER  AN 
IKTERRUPTION  AND  FOLLOWING  THROUGH;  PROVIDING  PILOTS  WITH  ADDITIONAL  SERVICES  AS  TIME  ALLOWS 


Has  difficulty  keeping  track  of  Keeps  on  top  of  movement  of  is  extremely  adept  at  keeping 

several  aircraft  at  the  same  time  several  aircraft  simultaneously  track  of  many  aircraft  while  at 

may  focus  too  narrowly  on  some  while  also  dealing  with  routine  the  same  time  handling  pilot 

aircraft  while  ignoring  others  communication.  When  very  communications,  strip  work, 

busy  may  have  to  simplify  the  etc. 
situation  to  reduce  the  number 
of  things  attended  to. 

Is  ineffective  at  performing  Is  good  at  performing  two  or  Is  fully  capable  of  performing 

multiple  tasks  simultaneously  sometimes  more  routine  tasks  two  or  more  complex  tasks 

prefers  to  take  one  thing  at  a  at  the  same  time  (e.g.  monitor-  simultaneously, 

time  Ing  the  screen,  talking  with  pi¬ 

lots  and  handling  strips) 

Interruptions  and  distractions  After  an  interruption,  can  usu-  interruption,  always 

often  cause  him/her  to  forget  ally  handle  the  air  traffic  pro-  quickly  remembers  where  air- 

about  some  of  the  Immediate  blems  remaining  from  prior  to  craft  are  or  should  be,  wjiat 

air  traffic  problems;  may  be  the  interruption  successfully.  doing  with  the 

slow  in  recalling  what  he/she  traffic  before  the  interruption, 

intended  to  before  the  interrupt  ^Dd  the  intended  control 

strategy _ 


1  23  4567  89  10 
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TECHNICAL  KNOWLEDGE 


KNOWING  THE  EQUIPMENT  AND  FTS  CAPABILITIES  AND  USING  IT  EFFECTIVELY;  KNOWING  AIRCRAFT  CAPABILITIES 
LIMITATIONS  (SPEED,  WAKE  REQUIREMENTS)  AND  USING  THAT  KNOWLEDGE;  KEEPING  UP-TO-DATE  ON  LETTERS 
OF  AGREEMENT,  CHANGES  IN  PROCEDURES,  REGULATIONS,  ETC.  KEEPING  UP-TO-DATE  ON  SELDOM  USED 
PROCEDURES  OR  SKILLS 


At  times,  may  not  remain  current 
on  new  letters  of  agreement, 
revised  air  traffic  procedures, 
etc. 


Has  basic  knowledge  of  most 
aircrafts*  capabilities,  but  may 
make  errors  related  to  not  know¬ 
ing  aircraft  limitations 


May  be  unfamiliar  with  some  of 
his/her  equipment  and  how  it 
works 


Is  usually  knowledgeable  about 
and  up-to-date  on  all  informa¬ 
tion  relevant  to  controlling 
traffic  (e.g.  letters  of  agree¬ 
ment,  air  traffic  procedures, 
etc.) 

Has  good  knowlege  of  different 
aircraft  capabilities  and  applies 
that  knowledge  to  avoid  most 
errors  associated  with  not 
knowing  aircraft  limitations. 


Is  reasonably  familiar  with  his/ 
her  eqipment  and  how  it  works 


Always  keeps  up-to-date  on 
letters  of  agreement,  ail  per¬ 
tinent  procedures  and  poli¬ 
cies,  any  sector-specific 
changes  (e.g,  revised  boun¬ 
daries) 

Has  thorough  knowledge  of 
different  aircraft  capabilities 
and  as  a  result  never  makes 
errors  such  as  climbing  an 
aircraft  beyond  its  limits,  mak¬ 
ing  an  inappropriate  speed 
assignment,  or  requiring  an 
impossibly  tight  turn. 

Is  extremely  knowledgeable 
about  and  familiar  with  his/ 
her  equipment  and  how  it 
functions  _ 
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REACTING  TO  STRESS 


REMAINING  CALM  AND  COOL  UNDER  STRESSFUL  SITUATIONS;  HANDLING  STRESSFUL  AIR  TRAFFIC 
CONDITIONS  IN  A  PROFESSIONAL  MANNER 


Becomes  shaken  and  Ineffec¬ 
tive  in  emergency  situations. 

Remains  calm  and  cool  to  most 
emergency  situations 

Remains  very  calm  and  cool 
and  reacts  effectively  even  In 
very  serious  emergency  sit¬ 
uations  such  as  aircraft  in¬ 
flight  emergencies,  lost  pilots 
etc. 

Reacts  poorly  and  performance 
suffers  under  stressful  air 
traffic  conditions 

Stays,  calm  ,  focused  and  func¬ 
tional  under  busy  conditions; 
may  be  somewhat  less  effective 
in  very  stressful  air  traffic  situa¬ 
tions 

Stays  calm,  focused  and  very 
functional  in  busy  and  very 
stressful  conditions 

Does  not  function  effectively 
when  equipment/system 
problems  arise 

Shows  professional  cool  in 
handling  routine  equipment/ 
system  problems 

Handles  even  serious  equip¬ 
ment/system  degradation 
probelms  with  professional 
cool 

1  23  4  567  89  10 
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MAINTAINING  ATTENTION  AND  VIGILANCE 


SCANNING  PROPERLY  FOR  AIR  TRAFFIC  EVENTS,  SflUATIONS,  POTENTIAL  PROBLEMS  ETC. 
KEEPING  TRACK  OF  EQUIPMENT  WEATHER  STATUS  ;  IDENTIFYING  UNUSUAL  EVENTS,  IMPROPER 
POSITIONING  OF  AIRCRAFT,  RECOGNIZING  WHEN  AIRCRAFT  HAVE  POTENTIAL  FOR  LOSS  OF 
SEPARATION;  VERIFYING  VISUALLY  THAT  CONTROL  INSTRUCTIONS  ARE  FOLLOWED,  REMAINING 
VIGILANT  DURING  SLOW  PERIODS 


Has  a  tendency  to  focus  too 
narrowly  on  one  air  traffic  pro¬ 
blem  and  sometimes  fails  to 
recognize  other  potential  pro¬ 
blems  with  conflictions,  traffic 
flow  etc. 

Often  does  not  recognize  that 
an  action  is  required;  is  often 
lax  In  watching  the  radar  scope 
and  tends  to  signficantly  reduce 
vigilance  during  slow  periods 


Has  problems  remembering  that 
an  action  was  taken  or  that  an 
action  is  required 


For  the  most  part,  properly 
scans  the  scope  and  monitors 
aircraft  to  maintain  awareness 
of  air  traffic  events,  potential 
problems  etc. 


Is  attentive  to  the  radar  scope 
and  maintains  vigilance,  espe¬ 
cially  during  rush  periods;  may 
sometimes  be  Inattentive  when 
traffic  Is  light 


Seldom  forgets  own  actions 
taken  or  that  an  action  is 
required 


Consistently  recognizes  po¬ 
tentially  dangerous  conditions 
such  as  errors  made  by  pilots 
(i.e.  wrong  turns,  descending 
through  assigned  altitude) 


Always  checks  and  verifies 
that  clearances  and  other  In¬ 
structions  to  pilots  are  fol¬ 
lowed;  remains  highly  vigilant 
even  during  slow  periods. 


Is  very  good  at  remembering 
own  actions  taken  or  that  an 
action  is  required  (I.e.  change 
of  course  to  avoid  restricted 
area) 


1  2  3 
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PRIORITIZING 


TAKING  EARLY  OR  PROMPT  ACTION  ON  AIR  TRAFFIC  PROBLEMS  RATHER  THAN  WAHTING  OR  GETTING  BEHIND 
KNOWING  WHAT  TO  DO  FIRST  AND  IDENTIFYING  THE  MOST  IMPORTANT  SITUATIONS;  RECOGNIZING  THAT  SOME  PROBLEMS 
OR  SITUATIONS  ARE  LESS  IMPORTANT  AND  CAN  WAIT;  PREPLANNING  BEFORE  BUSY  PERIODS;  ORGANIZING  THE  BOARD 
AND  USING  FLIGHT  STRIPS  EFFECTIVELY  TO  KEEP  PRIORITIES  STRAIGHT  FOR  HANDLING  AIR  TRAFFIC  SITUATIONS 
QUICKLY  AND  DECISIVELY  DETERMINING  APPROPRIATE  PRIORITIES. 


Has  difficulty  recognizing  which 
air  traffic  problems  are  the  most 
pressing;  may  deal  with  pro¬ 
blems  in  chronological  order,  or 
take  the  easy  ones  first. 


Usually  recognizes  the  most 
Important  air  traffic  problems 
and  handles  them  before  the 
less  pressing  ones 


Always  recognizes  which  air 
traffic  problems  need  Immedi¬ 
ate  attention  and  handles  then 
before  less  pressing  ones; 
recognizes  appropriate  priori¬ 
ties  for  control  actions 


Often  acts  on  air  traffic  problems  Normally  looks  ahead  to  assess 


without  evaluating  the  possible 
consequences  of  these  actions 


potential  air  traffic  problems 
that  might  result  from  own 
actions  or  from  changing  con¬ 
ditions. 


Is  very  good  at  looking  ahead 
to  assess  potential  problems 
that  might  result  from  revised 
clearances,  aircraft  counts  or 
altitude  changes 


Often  puts  off  decisions  or 
actions  that  should  be  taken 
right  away 


Is  usually  good  about  taking 
early  or  prompt  action  on  air 
traffic  problems;  may  some¬ 
times  put  off  a  decision  or  an 
aciton  that  should  be  attended 
to  immediately 


Consistently  takes  early  or 
prompt  action  on  air  traffic 
problems 
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MAINTAINING  SAFE  AND  EFFICIENT  TRAFFIC  FLOW 


REACTING  TO  AND  RESOLVING  POTENTIAL  CONFLICTIONS  EFFECTIVELY  AND  EFFICIENTLY;  USING  PROPER  AIR 
TRAFRC  SEPARATION  TE^^^^^  EFFECTIVELY  TO  ENSURE  SAFETY;  SEQUENCING  AIRCRAFT  EFFECTIVELY  FOR 

aSl  OR  DEPARTURE;  SEQUENCING  AIRCRAFT  TO  ENSURE  EFFICIENT/TIMELY  TRAFFIC  FLOW;  CONTROLUNG 
TRAFFIC  IN  A  MANNER  THAT  ENSURES  EFFICIENT  TRAFFIC  FLOW;  CONTROLLING  TRAFFIC  IN  A  MANNER  THAT 
MINIMIZES  TRAFFIC  PROBLEMS  (E.G.  CONFLICTIONS,  TRAFFIC  FLOW  PROBLEMS)  FOR  OTHER  CONTROLLERS  AND 
PILOTS 


Sometimes  fails  to  maintain 
minimum  separation  or  to 
recognize  and  resolve  poten¬ 
tial  conflictions 


Uses  control  actions  that  fail 
to  resolve  potential  conflictions 
or  that  result  In  excessive  work¬ 
load  (e.g.  waits  until  potential 
conflictions  are  critical  before 
taking  action 

Does  not  always  sequence 
aircraft  adequately  or  ensure 
proper  spacing  between  air¬ 
craft;  may  cause  excessive 
and  unnecessary  delays  by 
choosing  poor  control  actions 
waiting  too  long  to  provided 
needed  commands,  etc. 


Typically  uses  appropriate  con 
trol  actions  to  maintain  proper 
separation  or  to  resolve  poten¬ 
tial  conflictions 


Resolves  simple  conflictions 
and  traffic  flow  problems  quick¬ 
ly;  without  causing  unneces¬ 
sary  delays. 


Generally  uses  correct  proce¬ 
dures  to  sequence  and  space 
aircraft  safely;  maintains 
smooth  traffic  flow,  but  may 
not  use  the  most  efficient  con¬ 
trol  actions  (e.g.  may  not  al¬ 
ways  take  aircraft  types  into 
account 


Consistently  maintains  safe, 
efficient,  and  orderly  traffic 
flow,  even  under  difficult  or  un¬ 
usual  circumstances  (e.g. 
extremely  heavy  traffic) 

Recognizes  potential  problems 
or  conditions  early  and  takes 
appropriate  actions  to  main¬ 
tain  spearatlon  and  minimize 
inconvenience 


Sequences  and  spaces  traffic 
effectively  and  efficiently;  even 
when  extremely  busy;  always 
maintains  proper  separation 
while  minimizing  delays  (e.g. 
avoides  delaying  vectors  as 
appropriate,  use  flow  control 
procedures  when  necessary 
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ADAPTABILITY  AND  FLEXIBILITY 

REACTING  EFFECTIVELY  TO  DIFFICULT  EQUIPMENT  PROBLEMS,  CHANGES  IN  WEATHER,  TRAFFIC 
SITUATIONS,  ETC,  OR  TO  UNEXPECTED  ACTIONS  ON  THE  PART  OF  OTHER  CONTROLLERS  OR 
PILOTS;  USING  CONTINGENCY  OR  FALL-BACK  STRATEGIES  EFFECTIVELY  WHEN  UNFORSEEN/ 
UNANTICIPATED  AIR  TRAFFIC  PROBLEMS  EMERGE  OR  IF  FIRST  PLAN  DOESNT  WORK;  ASKING 
FOR  HELP  WHEN  ITS  NEEDED;  DEVELOPING/EXECUTING  INNOVATIVE  SOLUTIONS  TO  AIR  TRAFFIC 
PROBLEMS-  DEALING  EFFECTIVELY  WITH  SITUATIONS  FOR  WHICH  THERE  MAY  NOT  BE  CLEARLY 
PRESCRIBED  PROCEDURES,  SITUATIONS  WHICH  REQUIRE  NOVEL  THINKING;  ADAPTING  TO  EQUIPMENT 
UPDATES.  NEW  PROCEDURES  ETC. 


Does  not  adjust  well  to  unusual 
and  difficult  air  traffic  situations 


Rarely  displays  good  "fall-back'* 
strategies  for  dealing  with  unan- 
ticpated  air  traffic  problems 


Is  ineffective  at  handling  air 
traffic  situations  with  no  clear¬ 
ly  prescribed  procedures 


Is  usally  able  to  adapt  effective¬ 
ly  to  dihicult  situations  such  as 
rapidly  worsening  weather, 
equipment  problems  etc. 


Frequently,  but  not  always,  has 
effective  contingency  strategies 
for  unforseen  or  unanticipated 
air  traffic  problems  when  they 
arise 


For  the  most  part,  is  good  at 
handling  air  traffic  situations 
that  have  no  "textbook  ans¬ 
wers",  but  does  better  with  the 
more  routine  problems 


Reacts  very  effectively  to  com¬ 
plicating  events  and  difficult 
equipment  problems. 


Is  very  adept  at  using  effective 
contingency  or  "fall-back"  stra¬ 
tegies  when  unforseen  or  un¬ 
anticipated  air  traffic  problems 
arise 


Deals  very  effectively  with  air 
traffic  situations  where  there 
are  no  clearly  prescribed  pro¬ 
cedures 
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APPENDIX  D 


GENERIC  SECTOR  TRAINING  MANUAL 


FORWARD 


The  purpose  of  this  Training  Guide  is  to  develop  Standard  Operating  Procedures  and  define  radar 
position  responsibilities  for  GENERA  APPROACH  CONTROL  (FAA's  Generic  Sector)  along 
with  presenting  operating  guidelines  to  be  followed  while  controlling  traffic  in  both  the  ACY  and 
Genera  scenarios. 

Controllers  are  expected  to  be  familiar  with  the  provisions  contained  in  this  guide  as  they  pertain 
to  the  operational  requirements  of  the  Genera  Arrival  and  Departure  positions  and  to  exercise 
their  best  judgment  if  they  encounter  situations  not  covered  by  it. 

Since  this  is  a  two  part  experiment  and  one  half  of  the  study  deals  specifically  with  ACY  traffic,  it 
is  envisioned  that  controllers  will  adhere  to  current  procedures  contained  in  the  Standard 
Operating  Procedures  (SOPs)  Manual  (ACY  71 10.4)  while  controlling  ACY  traffic.  With  the 
exception  of  certain  automation  restrictions  as  noted  in  this  guide,  most  traffic  should  closely 
follow  SOP  patterns. 


D-1 


BACKGROUND 


Earlier  PERI/FAA  experimental  studies,  in  memory  related  areas,  have  confirmed  that  inexpensive 
ATC  simulation  systems  can  provide  a  vehicle  for  testing  controller  performance  on  a  limited 
basis. 

Using  PC-based,  off  the  shelf,  hardware  and  TRACON II,  an  ATC  software  program  that 
graphically  depicts  a  radar  approach  control  scope  display,  experimenters  were  able  to  investigate 
the  effects  of  two  memory  strategies  (planning  and  flight  strip  management)  on  ATC 
performance. 

Since  completion  of  the  initial  experiments  in  May,  1993,  the  FAA  Technical  Center  Human 
Factors  Laboratory  has  purchased  several  high  fidelity,  state  of  the  art,  console  based  simulation 
and  training  systems  from  UFA  Inc.  of  Lexington,  MA.,  the  makers  of  ATCoach.  These  systems 
came  equipped  with  a  22  inch,  high  resolution,  color  radar  display  monitor,  with  three  button 
trackball,  ARTS  IIIA  keyboard,  pseudo-pilot  position  and  offer,  as  an  option,  a  speaker 
independent,  voice  recognition,  pilot  response  communications  system. 

ATCoach's  simulation  system  is  configured  to  closely  replicate  both  terminal  and  en  route 
environments  including  control  panels  and  keyboards.  Approach  Control  airspace  and  En  Route 
sector  boundaries  can  be  constructed  to  align  with  map  overlays  in  use  at  FAA  facilities. 

Scenarios  that  closely  resemble  the  daily  traffic  flow,  including  STARs  and  SIDs,  can  be 
developed  using  air  traffic  data  collected  from  actual  flight  progress  strips  of  past  days  traffic. 
Currently  in  use  at  several  Universities,  the  ATCoach  system  has  been  primarily  programmed  to 
satisfy  basic  radar  training  requirements  with  the  emphasis  on  developing  a  college  level 
curriculum  that  meets  FAA  criteria.  ATCoach  provides  staff  personnel  with  adequate  information 
to  enable  instructors  to  closely  monitor  new  developmental  for  adherence  to  ATC  rules  and 
procedures  as  well  as  analyzing  the  future  controllers  decision  making  skills. 

This  record  keeping  process  is  exceptional  in  that  everything  that  transpires  during  a  given  traffic 
scenario  is  collected  and  stored  in  memory.  So  far,  its  use  has  been  mostly  in  an  academic  setting 
and  many  of  its  outstanding  features  have  not  been  explored.  This  state  of  the  art  system  was 
successfully  programmed  to  allow  experimentors  to  evaluate  specific  memory  variables  in  a 
recently  completed  memory  enhancement  experiment  (May  1994). 

GENERAL  METHOD 

PERI/FAA  memory  experiments  which  began  in  January  1994,  included  modifying  and 
intensifying  the  record  keeping  capabilities  of  the  ATCoach  data  processor.  Specific  variables 
were  created  to  measure  controller  on-the-job  performance  in  memory  and  performance  related 
tasks.  Scenarios  were  designed  to  test  memory  limitations  under  a  variety  of  circumstances. 
Memory  techniques  /strategies  and  aids  were  tested  from  two  standpoints;  first,  to  check  for 
usability  under  heavy  workloads,  and  second,  to  assess  the  aids  ability  to  effectively  increase 
productivity  without  draining  mental  resources. 

Results  gathered  from  past  experiments  coupled  with  data  derived  from  this  performance  study 
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should  prove  to  be  invaluable  in  evaluating  additional  memory  aids,  developing  new  and  enhanced 
aids,  assessing  individual  controller  techniques/strategies  as  they  relate  to  reducing  memory 
requirements,  improving  memory  recall,  increasing  memory  capacity,  and  hopefully,  in  measuring 
controller  performance. 

Additionally,  information,  collected  from  past  experiments,  is  being  used  as  the  basis  for  a 
database  that  should  enable  fiiture  researchers  to  detect  specific  variables  that  increase/decrease 
reliance  on  working  memory  and  increase/decrease  a  controllers  traffic  handling  capacities  while 
improving  performance.  This  data  will  be  indispensable  in  defining  individual  controller  memory 
limitations  and  recording  performance  as  it  relates  to  productivity. 

ATCOACH  SPECIFICS  and  SCENARIO  RESTRICTIONS 

ATCoach  is  pretty  particular  about  certain  separation  standards. 

YOU  MUST  STAY  AT  LEAST  THREE  MILES  FROM  THE  FINAL  APPROACH  COURSE 
WHEN  VECTORING  AN  AIRCRAFT  ON  DOWNWIND  LEG.  IN  MOST  CASES  IT  IS 
BEST  TO  USE  A  WIDER  DOWNWIND  LEG  OR  ALTITUDE  SEPARATION  TO 
PRECLUDE  A  "CA"  ALERT. 

YOU  MUST  USE  THE  TWENTY/THIRTY  DEGREE  INTERCEPT  RULES  WHEN 
TURNING  AN  AIRCRAFT  ON  THE  FINAL  INTERCEPT  HEADING  PRIOR  TO  ISSUING 
APPROACH  CLEARANCE.  ALLOW  AN  EXTRA  MILE  OR  TWO.  BETTER  YET,  MAKE 
ALL  YOUR  TURN  ONS  AT  TEN  MILES. 

TRY  TO  PREPLAN  ALL  OF  YOUR  ACTIONS.  SINCE  THIS  IS  A  COMPUTER  BASED 
SIMULATION,  IT  WILL  BE  EvlPOSSIBLE  TO: 

A)  MAKE  TIGHT  TURNS 

B)  EXPEDITE  CLIMBS/DESCENTS 

C)  MAKE  VISUAL  APPROACHES 

If  you  make  a  poor  turn  on  and  your  aircraft  executes  a  missed  approach,  you  will  still  be 
responsible  for  safely  vectoring  the  aircraft  back  around  for  another  approach.  If  the  turn  to  final 
is  the  fault  of  the  pseudo  pilot  or  computer,  the  aircraft  will  be  allowed  to  land  and  you  won't  be 
penalized.  This  decision  will  be  made  by  a  PERI  staff  controller  who  has  a  background  in  , 
terminals  and  will  be  monitoring  your  control  techniques. 

Flight  progress  strips  will  closely  resemble  those  currently  used  at  your  facility.  Controllers  are 
urged  to  use  them  as  a  back-up  system.  The  ARTS  1 1 1 A  keypack  should  enable  controllers  to 
perform  computer  functions  that  closely  replicate  their  facilities  environment.  Phraseology  should 
be  in  accordance  with  the  ATP  71 10.65  manual. 
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Although  no  provisions  were  made  for  separate  landline  communications,  coordination  should  be 
handled  in  a  manner  similar  to  facility  SOPs  simulating  the  use  of  an  intercom  for  voice 
communications.  The  pseudo-pilots  will  initiate  most  coordination  requests. 

The  ATCoach  simulator  has  been  preset  at  a  range  that  will  offer  optimum  radar  coverage  for  the 
scenarios.  You  may  want  some  adjustments  made  prior  to  testing.  We  can  alter  the  following: 

a)  range  mark  intensity 

b)  A/N  character  size 

c)  compass  rose-on\off 

d)  radar  range 

Efforts  have  been  made  to  keep  simulation  scenarios  realistic  and  challenging,  including  using 
actual  aircraft  call  signs  gathered  from  past  traffic  statistics.  Control  instructions  will  be  issued  to 
pseudo-pilots  who  will  make  the  necessary  computer  entries  and  try  to  respond  in  a  professional 
manner.  Since  it  is  virtually  impossible  to  cover  all  phraseology  associated  with  controlling  traffic 
in  a  terminal  environment,  you  may  have  to  repeat  some  instructions.  Your  patience  will  be 
appreciated. 

Pay  close  attention  to  readbacks.  Hopefully,  the  pseudos  have  enough  experience  to  correctly 
read  back  clearances  but  there  still  may  be  an  occasional  inverted  number.  If  a  mistake  is  made  it 
will  be  purely  unintentional  as  there  aren't  any  built-in  glitches. 

NOTE:  Standard  responsibilities  for  the  separation  and  control  of  Air  Traffic  are  not  reiterated  in 
this  guide,  since  they  are  explained  in  other  handbooks  and  manuals  and  controllers  should  be 
familiar  with  their  application. 

EXCEPTIONS: 

Due  to  the  experimental  design,  a  few  exceptions  were  made: 

a)  Controllers  have  to  accept  and  give  radar  hand-offs.  Departures  will  not  automatically 
hand-off. 

b)  All  radar  positions  are  combined.  Although  the  combining  of  positions  is  a  routine 
occurrence,  it  only  happens  under  light  to  moderate  traffic  conditions.  Most  of  the  heavy 
traffic  scenarios  would  require  de-combining  the  positions.  Keeping  positions  combined  was 
a  intentional  manipulation  designed  to  increase  complexity  by  forcing  controllers  to  scan  the 
entire  scope,  thereby,  increasing  their  workload. 

**  NOTE:  The  heavy  scenarios  are  deliberately  designed  to  replicate  an  overloaded  terminal 
sector.  We  are  trying  to  determine  when  a  sector  is  saturated  using  a  combination  of  heavy 
volume  and  traffic  complexity.  Do  not  feel  intimidated.  You  are  expected  to  stop  the  traffic  flow 
and  initiate  holding  whenever  you  see  the  need. 
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c)  Controllers  will  be  required  to  make  final  approach  turn  ons  to  all  satellite  airports.  In  order  to 
equalize  workload,  no  provisions  were  made  for  using  the  Rainbow  Transition  or  other  automated 
approach  route  segments. 

Scenario  Design  ACY 

Scenarios  were  designed  to  take  into  account  most  of  the  tasks  associated  with  controlling  the 
Atlantic  City  South  Arrival  and  Departure  radar  positions,  in  a  South  landing  configuration  at  the 
Atlantic  City  Approach  Control  under  Instrument  Flight  Rules  (IFR).  For  all  of  the  test  scenarios, 
we  have  programmed  the  following  weathers. 

ACY  :  W4X1F  56/49  1508  30.00  ILS  Runway  13  approaches  in  use 

AIY  :  W7X2F  54/50  1408  30.00  VOR  Runway  1 1  approaches  in  use 

MIV  :  W6X1F  55/48  1205  30.00  ILS  Runway  10  approaches  in  use 

WWD  ;  W8X2F  57/51  1606  30.00  ILS  Runway  19  approaches  in  us 

Information  "A"  is  the  current  AXIS. 

The  weather  dictates  that  all  scenario  aircraft  have  to  be  treated  as  instrument  flights.  This 
precludes  using  any  Visual  Approach  techniques.  Scenarios  consist  of  light  and  heavy  traffic 
volumes  and  will  be  assigned  at  the  discretion  of  the  experimenters.  Holding  patterns  have  been 
established  which  correlate  with  airspace  boundaries. 

Scenario  Design  Genera  Approach 

Scenarios  were  designed  to  take  into  account  most  of  the  tasks  associated  with  controlling  the 
Genera  North  Arrival  and  Departure  radar  positions,  in  a  North  landing  configuration  at  Genera 
Approach  Control  under  Instrument  Flight  Rules  (IFR).  For  all  of  the  test  scenarios,  we  have 
programmed  the  following  weathers. 

GEN  :  W5X2F  56/49  3608  30.00  ILS  Runway  36R  approaches  in  use 
DWN  :  W6X1F  55/48  0205  30.00  ELS  Runway  04  approaches  in  use 
MTD  ;  W7X2F  54/50  3408  30.00  ELS  Runway  33  approaches  in  use 
UPT  :  W8X2F  57/5 1  3606  30.00  ILS  Runway  01  approaches  in  use 
Information  "G"  is  the  current  AXIS. 

The  weather  dictates  that  all  scenario  aircraft  have  to  be  treated  as  instrument  flights.  This 
precludes  using  any  Visual  Approach  techniques.  Scenarios  consist  of  light,  moderate  and  heavy 
traffic  volumes  and  will  be  assigned  at  the  discretion  of  the  experimenters.  Holding  patterns  have 
been  established  which  correlate  with  airspace  boundaries. 
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ARRIVAL  AND  DEPARTURE  CONTROL 


Controllers  are  expected  to  comply  with  facility  established  routings  as  depicted  in  the  Letters  of 
Agreement  (LOAs). 

1 .  Arrival  Control —  Arrivals  should  be  vectored  and  descended  within  the  confines  of  the  Arrival 
Airspace  Jurisdiction  Charts  as  depicted  herein. 

a)  Hand-offs  should  be  accepted  as  soon  as  the  aircraft  are  flashing  in  a  hand-off  mode. 

b)  Although  positions  are  combined,  all  efforts  should  be  made  to  conform  to  altitude 
restrictions  as  depicted  in  the  Airspace  Jurisdiction  charts. 

2. )  Departure  Control —  Departures  should  be  vectored  and  climbed  within  the  confines  of  the 
Departure  Airspace  Jurisdiction  Chart  as  depicted  herein. 

a)  Hand-offs  should  not  be  made  until  the  departing  aircraft  is  clear  of  all  traffic,  climbing  to 
prescribed  altitude,  and  navigating  to  first  assigned  fix  outside  of  Genera  Airspace. 

b)  Although  positions  are  combined,  all  efforts  should  be  made  to  conform  to  altitude 
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APPENDIX  E 

FINAL  QUESTIONNAIRE  RESPONSES 

1.  How  realistic  was  the  simulation? 

01.  Very  good,  other  than  the  DH8  descent. 

02.  The  simulation  was  very  realistic. 

03.  No  response. 

04.  Very. 

05.  For  the  most  part  very  realistic. 

06.  This  simulation  was  very  realistic.  It  standardized  the  arrival  and  departure  sequence 
allowing  me  to  accept  more  traffic  without  losing  sight  of  my  objective. 

07.  Very  realistic,  better  than  what  we  use  in  our  TTG  simulator. 

08.  Pretty  realistic. 

09.  Very  real. 

10.  About  90%  realistic,  work  still  needs  to  be  done  on  descent  rates  and  turn  rates. 

1 1 .  The  simulation  seemed  very  realistic. 

2.  How  representative  was  the  generic  sector  of  a  typical  terminal  environment? 

01.  OK,  maybe  too  basic. 

02.  Very  representative,  except  that  ACY  normally  works  overflight  traffic. 

03.  No  response. 

04.  Very. 

05.  N/A  (only  worked  ACY). 

06.  Very  representative  of  a  level  V  type  approach  control. 

07.  Fairly  well  representative,  satellite  airports  and  altitude  restrictions  seem  realistic. 

08.  Set  up  well. 

09.  Very  typical 

10.  Not  to  typical,  it  works  to  well. 

1 1 .  It  differs  much  from  ACY  operation,  but  I  suppose  it  is  typical  of  a  level  5  operation. 

3.  How  helpful  was  the  training  booklet  in  learning  the  generic  sector? 

01.  Didn't  look  at  it  before  the  problems. 

02.  Not  much  for  myself 
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03.  No  response. 

04.  Didn't  really  see  it. 

05.  Not  very,  too  much  information. 

06.  The  booklet  gave  me  enough  information  to  allow  me  to  begin  each  scenario  with 
minimal  required  information. 

07.  Somewhat,  there  is  more  information  than  is  needed,  however  the  airspace,  sector 
boundaries,  etc.,  are  very  helpful  to  know  in  advance. 

08.  Very  helpful. 

09.  Not  very. 

10.  More  information  in  the  booklet  than  was  needed.  It  could  have  been  simplified. 

11.  Good  to  learn  frequencies  and  airspace  layout,  but  I  didn't  really  understand  much  of  the 
operation  till  I  saw  it  in  operation. 

4.  Was  the  hands-on  training  adequate  on  the  day  1  session? 

01.  Yes,  very  good. 

02.  Yes. 

03.  No  response. 

04.  Yes!  Very. 

05.  Yes. 

06.  The  hands-on  training  provided  the  other  half  of  the  necessary  information  to  allow  me  to 
operate  with  confidence  and  ease. 

07.  Yes. 

08.  Yes. 

09.  Yes. 

10.  Yes. 

11.  Yes. 

5.  How  could  the  generic  sector  be  improved? 

01.  For  people  to  learn  it,  it's  pretty  good. 

02 

a)  Make  it  easier  to  move  leader  lines. 

b)  Create  a  little  more  space  for  strips. 

c)  Have  a  foot  pedal. 

d)  The  airspace  is  fine. 
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03.  No  response. 

04.  Freq's  N/E-odd  (ncc)  135.0/(ncc)  137.0SAV-even  (see)  136.0/(see)  138.0 
05.  Blank. 

06. 1  do  not  think  it  would  need  any  further  improvement  at  this  time. 

07.  Add  in  some  overflights. 

08.  Transfer  of  departures  eould  be  sooner. 

09.  Need  a  foot  pedal. 

10.  No  improvements  needed. 

1 1. 1  don't  know  how  you  eould  take  a  unfamiliar  seetor  and  make  it  any  easier  than  it  is. 
The  standard  climbs  and  descents  made  it  very  easy  along  with  names  (intersections,  arps, 
etc.)  and  the  north,  south  setup. 

6.  Did  the  ATWIT  device  interfere  with  controlling  traffic  on  either  sector? 

01.  No. 

02.  No.  I  would  disregard  it  for  a  second  or  two  if  I  had  a  priority  to  work  on. 

03.  No  response. 

04.  No. 

05.  No. 

06.  No. 

07.  No,  we  used  the  same  equipment  in  the  previous  project  and  it's  fairly  automatic  now,  I 
didn't  really  even  notice  that  I  was  responding. 

08.  No. 

09.  No. 

10.  No. 

11.  No. 

7.  How  well  did  the  pseudo-pilots  respond  to  your  clearances  in  terms  of  traffic  movement  and 
call-backs? 

01.  Fair,  they  we're  new. 

02.  They  were  good  except  for  1  problem. 

03.  No  response. 

04.  Very  well,  excellent. 

05.  Pseudo-pilots  did  a  very  good  job. 

06.  Excellent 


07.  Very  well,  they  did  an  excellent  job.  ■ 

08.  Response  was  very  good. 

09.  Good  job. 

10.  95%  correct. 

1 1.  Very  good  job. 

Do  you  have  any  other  comments  about  your  experiences  during  the  simulation? 

01. 1  enjoyed  this  one. 

02.  No. 

03.  No  response. 

04. 1  enjoyed  it  a  great  deal! 

05.  Blank. 

06.  No. 

07.  Just  that  I'm  anxious  to  see  the  results! 

08.  Blank. 

09.  No.  Nice  system. 

10.  Having  intersections  10  miles  from  the  0/M  on  all  ILS  apch's  helped  greatly.  You  do 
need  a  foot  pedal  with  all  the  writing  and  strip  management  that  is  necessary! 

11.  No. 
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